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CHAPTER  I 


INTRODUCTION 

Background 

Test  and  evaluation  (T&E)  of  new  weapon  systems  and 

equipment  is  an  integral  part  of  the  acquisition  process. 

T&E  is  used  in  the  system  acquisition  process 

...  to  Identify,  assess,  and  reduce  the  acqui¬ 
sition  risks,  to  evaluate  operational  effectiveness 
and  operational  suitability,  and  to  identify  any 
deficiencies  in  the  system  [17:2]. 

Air  Force  T&E  consists  of  three  types:  Development  Test 

and  Evaluation  (DT&E),  Qualification  Test  and  Evaluation 

(QT&E),  and  Operational  Test  and  Evaluation  (OT&E). 

Development  Test  and  Evaluation 

Development  Test  and  Evaluation  is  the  responsibil¬ 
ity  of  the  Implementing  command.  DT&E  is  basically  an  in- 
depth  engineering  analysis  of  a  system's  performance  where 
the  system  design  is  tested  and  evaluated  against  engineer¬ 
ing  and  performance  criteria.  The  testing  begins  with  indi¬ 
vidual  subsystems,  or  components,  and  progresses  up  to 
prototypes  of  the  entire  system.  Modifications  to  the  sys¬ 
tem,  even  after  production,  must  undergo  DT&E;  therefore, 
DT&E  might  be  conducted  throughout  the  life  of  the  system 
(17:2). 
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The  objectives  of  DT&E  as  outlined  In  APR  80-lA  are: 


a.  To  assess  the  critical  Issues,  as  specified  in 
program  documents; 

b.  To  assess  the  technical  risk  and  evaluate  the 
trade-offs  among  specification,  operational  require¬ 
ments,  life  cycle  costs,  and  schedules; 

c.  To  verify  the  accuracy  and  completeness  of  the 
technical  orders  developed  to  maintain  and  operate  the 
weapons  system; 

d.  To  gather  Information  for  training  programs  and 
technical  training  materials  needed  to  support  the 
weapon  system; 

e.  To  provide  Information  on  environmental  Issues 
to  be  used  In  preparing  environmental  Impact  assess¬ 
ment  s ; 


f.  To  determine  system  performance  limitations 
and  safe  operating  parameters  [17:6]. 


Qualification  Test  and  Evaluation 

Qualification  Test  and  Evaluation  (QT&E)  Is  per¬ 
formed  Instead  of  DT&E  on  programs  where  there  are  no  funds 
In  the  research,  development,  test,  and  evaluation  appro¬ 
priation.  QT&E  Is  performed  by  the  Implementing  command. 
The  test  policies  for  DT&E  also  apply  to  QT&E  (17:3).  QT&E 
Is  not  as  widely  known  and  discussed  as  Its  sisters  DT&E 
and  OT&E. 


Operational  Test  and  Evaluation 

AP  Operational  Test  and  Evaluation  is  managed  by 
the  Air  Force  Test  and  Evaluation  Center  (APTEC).  OT&E  is 
the  test  and  evaluation  to  determine  the  expected  operation¬ 
al  effectiveness  and  operational  suitability.  It  is 


conducted  throughout  the  system's  life  cycle  beginning  with 
Initial  Operational  Test  and  Evaluation  (lOT&E)  in  the  pre- 
production  phases  and  Follow-on  Operational  Test  and  Evalu¬ 
ation  (FOT&E)  after  the  production  decision  is  made.  OT&E 
is  to  be  conducted  in  an  environment  for  which  the  system 
is  Intended.  This  includes  the  use  of  personnel  with  the 
same  types  of  skills  and  qualifications  as  those  who  will 
operate,  maintain,  and  support  the  system  when  it  is 
deployed  (17:2). 

The  objectives  of  OT&E  as  outlined  in  AFR  80-14  are 

a.  To  estimate  the  operational  effectiveness  and 
suitability  of  the  system; 

b.  To  Identify  operational  deficiencies; 

c.  To  recommend  and  evaluate  changes  in  produc¬ 
tion  configuration; 

d.  To  provide  information  for  developing  and 
r€  fining: 

(1)  Logistics  support  requirements  for  the 

system, 

(2)  Training,  tactics,  techniques,  and  doc¬ 
trine  throughout  the  life  of  the  system; 

e.  To  provide  information  to  refine  operation  and 
support  (O&S)  cost  estimates,  and  to  identify  system 
characteristics  of  deficiencies  that  can  significantly 
impact  O&S  costs; 

f.  To  find  out  whether  the  technical  publications 
and  support  equipment  are  adequate; 

g.  To  estimate  the  survivability  of  the  system  in 
the  operational  environment  [17:6]. 


Responsible  Test  Organizations 


Air  Force  Systems  Command  Deputy  Chief  of  Staff  for 


Test  and  Evaluation  (HQ  APSC/TE)^  is  required  to  appoint  a 
Responsible  Test  Organization  (RTO)  during  the  early  phase 
of  each  new  DT&E  program  (3:^).  The  RTO  Is  responsible  for 
accomplishing  the  test  program  as  approved  by  the  program 
manager.  The  RTO  provides  the  program  office  and  HQ  AFSC/ 
TE  " .  .  •  an  Independent  assessment  of  the  DT&E  results  in 
conjunction  with  program  decision  points  [3  •■6]."  The 
responsibilities  of  the  RTO  include  the  design  of  the  test 
program,  the  collection  and  reduction  of  test  data,  and  the 
arrangement  and  coordination  of  the  support  of  all  partici¬ 
pating  organizations  (1:^-5). 

The  RTO  is  normally  assigned  by  TE.  TE  examines 
the  mission  statements  of  available  Test  and  Evaluation 
(T&E)  organizations  to  assess  the  test  organizations'  capa¬ 
bilities  to  accomplish  the  required  testing.  Unique  test 
requirements  and  other  significant  factors  are  considered 
before  the  decision  is  made  to  assign  a  test  organization 
as  the  RTO.  The  test  organizations  considered  qualified 
are  sent  messages  informing  them  of  their  possible  assign¬ 
ment  as  RTO.  The  T&E  organizations  respond  to  TE’s  message 
with  pertinent  plans  for  the  upcoming  test  period.  TE 
reviews  the  responses  received  from  the  test  organizations 
before  making  the  final  decision.  The  test  organization 

^The  organization  will  be  referred  to  as  "TE" 
throughout  the  thesis. 


chosen  as  the  RTO  is  then  notified  by  message.  The  RTO  is 
designated  on  the  AFSC  Form  56,  AFSC  Program  Direction, 
which  is  forwarded  to  the  program  office  (9;  12). 


Test  Program  Development 

In  a  typical  acquisition  program,  the  Implementing 
command  assigns  an  acquisition  program  to  a  System  Program 
Office  (SPO).  A  program  manager  is  assigned  to  the  program 
to  manage  the  entire  system  acquisition  process  (except 
for  OT&E).  The  SPO  normally  contains  a  test  office  which 
supports  the  program  manager  by  providing  day  to  day  test 
management;  and  the  RTO,  the  user,  AFTEC,  and  other  support¬ 
ing  organizations  develop  the  Test  and  Evaluation  Master 
Plan  (TEMP).  The  TEMP  identifies  critical  program  issues 
to  be  addressed  in  the  test  and  evaluation  program.  The 
TEMP  also  states  how  the  test  will  be  conducted  and  how  the 
results  will  be  used  to  verify  the  stated  requirements 
(1:1-5;  3:5). 

Simulator  Testing 

The  Deputy  for  Simulators  (YW)  in  the  Aeronautical 
Systems  Division  (ASD)  is  responsible  for  development  of 
devices  to  be  used  to  train  personnel  in  AF  weapon  systems 
operational  procedures.  The  Director  of  Simulator  Test 
and  Deployment  (YWT)  has  the  responsibility  to: 

Develop  Quallflcatioti  Test  and  Evaluation  (QT&E) 
and  Development  Test  and  Evaluation  (DT&E)  concepts 
to  ensure  acquisition  of  simulation  devices  with 


characteristics  comparable  to  the  weapon  system  being 
simulated.  Establishes  procedures  for  consolidating 
DT&E  and  operational  test  requirements.  Establishes 
and  maintains  procedures  for  processing  and  tracking 
test  discrepancies  and  service  reports  for  all 
simulator  programs.  Develops  concepts  and  policies 
to  ensure  that  adequate  facility  planning,  as  well 
as  pre-operatlonal  support  are  accomplished  to  ensure 
the  development  of  simulation  devices  to  all  intended 
sites  [2:p.9-2]. 

The  Deputy  for  Simulators  has  historically  been 
assigned  as  the  RTO  for  simulator  test  programs.  This  came 
about  because  of  the  perceived  "uniqueness"  of  the  simula¬ 
tor  business.  When  YW  was  first  formed,  it  was  primarily 
a  one  program  office.  Its  main  function  at  that  time  was 
the  acquisition  of  Undergraduate  Pilot  Training  (UPT)  simu¬ 
lators  for  the  training  of  Air  Force  pilots.  As  sophisti¬ 
cation  of  Air  Force  aircraft  increased  and  the  cost  of 
flying  time  increased,  the  need  for  ground  simulators  also 
increased.  The  lack  of  AF  test  organization  expertise  in 
computer  software  during  the  early  days  of  simulator  test¬ 
ing  led  to  high  reliance  on  contractor  testing. 

Currently  simulator  testing  is  accomplished  at  the 
contractor's  facility  using  contractor  generated  test  pro¬ 
cedures.  The  test  plan  and  procedures  are  also  contractor 
prepared  documents.  Simulator  personnel  either  provide 
inputs  to  the  plans  or  Just  review  them  depending  on  the 
contractor's  experience  or  expertise.  Initial  system  test¬ 
ing  is  accomplished  by  the  contractor  to  verify  that  proper 
procedures  result  in  proper  indications  such  as  pushing  the 
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throttle  forward  causes  Increased  RPM  on  the  engine  RPM 
indicator.  The  contractor  also  verifies  the  software  pro¬ 
gram  during  this  phase  of  testing  when  it  is  required. 

Test  personnel  from  YWT  are  then  sent  TDY  to  the  contractor's 
facility  to  execute  the  test  plan.  Any  subsequent  problems 
with  the  simulator  may  require  additional  visits  until  test¬ 
ing  is  satisfactorily  completed.  The  simulator  is  then 
shipped  to  the  user's  location  where  acceptance  testing  is 
performed  by  YWT  personnel  to  verify  that  the  system  works 
after  installation.  YWT  personnel  write  the  simulator  test 
specifications,  manage  the  development  of  the  test  plan, 
and  then  actually  conduct  the  test  (5). 

Problem  Statement 

The  assignment  of  the  Directorate  of  Simulator 
Test  and  Deployment  (ASD/YWT)  as  the  RTO  provides  four 
potential  areas  of  concern.  First,  the  RTO  is  charged  with 
providing  "...  the  program  office  and  HQ  AFSC/TE  an  inde¬ 
pendent  assessment  of  the  DT&E  results  [3:6]."  With  ASD/ 

YWT  acting  as  the  RTO,  the  independent  assessment  may  be 
lost  because  of  the  functional  relationship  of  YWT  with  the 
SPO.  AFSC  guidance  for  assignment  of  RTO's  also  states 
that  care  must  be  taken  to  ".  .  .  avoid  making  the  SPO  the 
Responsible  Test  Organization  .  .  .  make  maximum  use  of 
test  and  user  organizations  in  support  of  SPO  [15]."  The 
independent  test  organization  provides  an  objective 
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perspective  of  the  test  program  (9)c 

In  a  normal  test  program  the  test  support  person¬ 
nel  in  a  SPO,  such  as  YWT,  are  concerned  with  overall  test 
planning,  identification  of  critical  issues,  coordination 
of  all  test  participants'  activities,  consultation,  and 
documentation  of  the  test  program.  The  RTO  is  a  separate 
organization  whose  personnel  are  concerned  with  designing 
and  conducting  the  actual  test  program.  Assignment  of  RTO 
responsibilities  to  YWT  was  not  accompanied  by  an  increase 
in  YWT  manning.  Therefore,  the  potential  exists  for  both 
test  management  and  RTO  responsibilities  to  suffer  because 
of  insufficient  manning  (4). 

Finally,  when  a  program  office  deals  with  an  out¬ 
side  RTO  a  contract-like  situation  exists.  The  program 
office  sends  the  RTO  a  Program  Introduction  Document  (PID) 
which  describes  the  test  support  needed.  The  RTO  sends 
back  to  the  program  office  a  Statement  of  Capability  (SOC) 
which  expresses  the  test  agency's  capability  to  support 
the  test  program.  A  Memorandum  of  Agreement  (MOA),  AFSC 
Form  607,  is  signed  by  the  program  office  and  the  test 
facility.  This  document  signifies  agreement  on  the  type 
and  level  of  support  to  be  provided  by  the  test  facility 
as  well  as  the  cost  of  that  support.  However,  with  the  RTO 
in  the  program  office  this  type  of  contractual  agreement 
has  been  absent.  Funds  designated  for  testing  can  be  more 
easily  be  diverted  into  the  general  program  fund  and  used 
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for  non-test  purposes.  Therefore,  having  the  RTO  in  the 
program  office  facilitates  shortening  or  deleting  tests; 
this  may  be  detrimental  to  the  overall  test  program. 

The  Director  of  Simulator  Test  and  Deployment  pro¬ 
posed  that  the  concerns  stated  above  are  reason  to  re¬ 
examine  the  assignment  of  the  YWT  as  the  RTO  for  simulator 
DT&E.  It  Is  possible  that  self-RTO  may  be  the  best  method 
for  simulator  DT&E;  however,  examination  of  the  concerns 
stated  may  indicate  a  need  to  create  an  organization  capa¬ 
ble  of  acting  as  an  independent  RTO  for  simulator  DT&E.  It 
may  also  be  possible  that  the  original  lack  of  simulator 
testing  expertise  in  AFSC  test  organizations,  which  led  to 
the  initial  assignment  of  YW  as  the  RTO,  may  not  be  the  case 
today.  There  may  be,  in  fact,  other  organizations  capable, 
or  potentially  capable,  of  acting  as  the  RTO  for  simulator 
testing. 


Research  Ob.jective 

The  objective  of  this  research  project  is  to  deter¬ 
mine  the  feasibility  of  assigning  an  independent  Responsi¬ 
ble  Test  Organization  for  development  test  and  evaluation 
of  Air  Force  simulators. 

Research  Questions 


In  order  to  accomplish  the  research  objective,  the 


following  questions  have  been  developed  to  guide  the  re¬ 
search  effort. 

1.  What  are  the  criteria  for  success  in  a  simula¬ 
tor  program' s  DT&E? 

a.  What  characteristics  of  simulator  DT&E  make 
it  "unique"  with  respect  to  other  testing  programs? 

b.  What  special  resources  are  required  to 
accomplish  simulator  DT&E? 

2.  Do  potential  or  real  problems  with  objectivity, 
control  of  test  funds,  and  manning  indicate  a  need  for  an 
Independent  RTO? 

a.  If  the  problems  do  exist,  can  the  problems 
of  self-RTO  be  lessened  or  eliminated  within  the  self-RTO 
structure? 

b.  If  the  problems  do  exist,  does  assignment 
of  an  independent  RTO  provide  a  better  means  of  lessening 
or  eliminating  the  problems? 

3.  If  a  need  exists  for  an  outside  test  agency 
for  simulator  DT&E,  is  there  a  test  organization  capable, 
or  potentially  capable,  of  acting  as  an  independent  RTO  for 
all,  or  part,  of  simulator  testing? 


10 


CHAPTER  II 


METHODOLOGY 

Introduction 

The  research  was  conducted  in  three  phases: 

(1)  determination  of  a  conceptual  model  for  simulator  test¬ 
ing,  (2)  data  collection,  and  (3)  data  analysis.  Collec¬ 
tively,  these  phases  comprised  the  research  methodology  and 
provided  the  basis  in  which  to  answer  the  research  questions 
and  satisfy  the  research  objective. 

Unique  Testing  Characteristics 

The  first  research  question  was  answered  by  defin¬ 
ing  the  simulator  DT&E  effort.  An  effective  DT&E  program 
for  simulator  testing  was  defined  as  meeting  the  basic 
objectives  of  DT&E  set  forth  in  APR  80-14  (17:6).  In  addi¬ 
tion,  it  was  necessary  to  define  the  characteristics  of 
simulator  testing  which  set  it  apart  from  other  DT&E  test¬ 
ing  programs.  Conversations  with  YW  personnel  involved  in 
simulator  testing  identified  the  ?-l6  Plight  Simulator  as 
a  typical  simulator  test  program  (5;  10).  Examination  of 
the  P-l6  simulator  program  provided  four  test  areas  which 
may  not  be  present  in  non-simulator  test  programs. 


Computational  system  tests  are  done  to  verify  that 
hardware  and  software  comply  with  design  specifications. 

They  involve  the  interaction  of  the  software  and  hardware 
Tests  are  conducted  to  demonstrate  that  the  hardware  inter¬ 
acts  wltn  the  computer  to  provide  an  accurate  simulation  of 
the  aircraft's  characteristics. 

Instructional  Systems 

Since  the  simulator  is  used  as  a  training  device, 
it  necessarily  has  provisions  for  Instruction  which  must  be 
tested  for  specification  compliance.  The  instructor  must 
be  able  to  interact  with  the  total  simulator  system  in  order 
to  control  and  monitor  the  simulator  mission.  For  example, 
cockpit  values  must  be  changeable  by  the  instructor  allow¬ 
ing  simulated,  inflight  malfunctions  to  be  given  to  the 
student  to  recognize  and  overcome. 

Mechanoreceptor  Cueing  Systems 

An  added  dimension  incorporated  in  aircraft  simula¬ 
tion  is  the  capability  to  generate  sensations  of  motion. 

For  example,  this  may  be  accomplished  by  applying  pressure 
to  various  parts  of  the  student's  body  through  various 
devices  in  or  on  the  seat  of  the  si.mulator.  Activation  of' 
the  system  through  movements  of  the  flight  controls  provide 


realistic  flight  sensations.  Just  as  any  system,  the 


mechanoreceptor  cueing  system  must  be  checked  for  specifi¬ 
cation  compliance. 


Visual  Systems 

Many  aircraft  simulators  provide  a  visual  represen¬ 
tation  of  the  flight  environment.  The  sophistication  of 
the  visual  representation  can  vary  from  program  to  program 
and  each  system  must  be  tested  for  compliance  to  the  design 
specifications . 

QT&E 

In  addition  to  the  four  testing  areas  which  may  be 
considered  unique,  simulator  DT&E  is  many  times  accomplished 
under  the  label  of  QT&E  because  a  program  has  not  been 
funded  in  the  research,  development,  test,  and  evaluation 
appropriation.  However,  the  test  policies  for  DT&E  and 
QT&E  are  the  same.  The  only  difference  between  the  terms 
is  the  source  of  test  funds  (17:3) •  For- purposes  of  this 
research  project,  a  distinction  was  not  made  between  DT&E 
and  QT&E.  Test  programs  with  objectives  as  described  for 
DT&E  in  APR  80-1^  (17:6)  were  referred  to  under  the  general 
classification  of  DT&E. 

Resource  Requirements 

With  the  unique  characteristics  of  simulator  test¬ 
ing  defined,  the  next  step  was  to  determine  the  special 
resources  required  to  perform  simulator  testing  in  each  of 


the  four  unique  areas.  The  F-l6  Plight  Simulator  Trainer 
was  used  as  a  typical  aircraft  simulator  test  program.  The 
test  plan  (13)  for  the  F-16  simulator  was  examined  to  deter¬ 
mine  the  types  of  test  personnel  required  to  perform  the 
testing  in  each  of  the  four  areas.  With  the  resource 
requirements  in  mind,  interviews  were  conducted  with  person¬ 
nel  currently  involved  in  each  of  the  four  areas  peculiar 
to  simulator  testing  to  determine  what  criteria  are  used  to 
assign  personnel  to  a  test  team  for  each  of  the  areas.  For 
example,  when  a  test  plan  calls  for  a  "qualified  computer 
person,"  what  criteria  are  used  in  the  determination  of  a 
"qualified"  person?  Are  these  qualifications  standard  for 
most  programs  or  do  they  vary?  Once  these  qualifications 
were  determined  for  each  of  the  "unique"  simulator  testing 
areas,  it  was  possible  to  determine  what  portion  of  the 
simulator  test  program  could  be  accomplished  by  personnel 
outside  of  the  YW  organization. 

RTO  Survey 

The  second  research  question  was  answered  by  sur¬ 
veying  test  managers  involved  in  test  programs  where  the 
program  office  was  assigned  FTO  responsibilities.  The  thrust 
of  this  survey  was  directed  at  determining  whether  problems 
exist,  or  could  exist,  with  regard  to  objectivity,  control 
of  test  funds,  and  manning.  Test  managers  were  also  asked 
whether  an  independent  RTO  could  increase  the  effectiveness 
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of  testing  with  regard  to  these  problems,  or  whether  the 
oroblems  could  be  rectified  within  the  self-RTO  structure. 


In  addition,  the  test  managers  were  surveyed  concerning  the 
factors  which  were  used  to  determine  their  status  as  a  self- 
RTO. 

Population 

Because  the  majority  of  AF  acquisition  programs  are 
accomplished  by  AFSC,  AFSC  was  the  only  implementing  command 
considered  as  a  source  of  data.  In  addition,  AFSC  was 
further  limited  to  three  of  its  maj'or  divisions:  Aeronau¬ 
tical  Systems  Division  (ASD);  Electronic  Systems  Division 
(ESD);  and  Space  Division  (SD).  After  conversations  with 
HQ  AFSC/TEV  (9)  concerning  AFSC  DT&E  and  RTO  assignment,  it 
was  determined  that  these  major  divisions  contained  the 
total  population  of  organizations  Involved  in  DT&E  under 
the  self-RTO  concept.  A  current  list  of  programs  under 
self-RTO  was  obtained  from  ASD,  ESD,  and  SD  with  designated 
points  of  contact  for  test  and  evaluation.  Initially,  the 
points  of  contact  were  used  to  determine  the  current  status 
of  their  program  and  the  test  manager's  name.  A  question¬ 
naire  was  administered  over  the  telephone  to  the  person 
designated  as  a  test  manager  for  DT&E  for  the  program.  Pro¬ 
gram  managers  who  were  also  acting  as  the  test  manager  were 
not  surveyed  because  portions  of  the  survey  addressed  possi¬ 
ble  conflicts  between  the  program  manager  and  test  manager. 
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Twelve  AFSC  program  offices  were  found  performing 
DT&E  with  the  test  manager  located  in  the  program  office. 
These  programs  were  found  in  ESD  and  SD.  YW  was  the  only 
program  in  ASD  performing  DT&E  with  test  managers  in  the 
program  office.  The  limited  number  of  self-RTO  programs 
provided  a  relatively  small  population;  this  small  number 
did  allow  an  exhaustive  survey  rather  than  a  sample. 

Survey  Construction 

The  self-RTO  survey  (see  Appendix  A)  was  used  to 
determine  how  AFSC  DTScE  programs  operate  under  the  self-RTO 
concept.  The  questions  were  designed  to  address  six  general 
areas ; 

1.  demographic  information  (1,  2,  30>  31); 

2.  manning  requirements  (3  -  8,  24); 

3.  control  of  program  test  funds  (9  -  16,  23); 

4.  program  objectivity  (17,  18,  23); 

5.  RTO  assignment  process  (19  -  22,  25,  27,  28);  and 

6.  suggested  improvements  to  self-RTO  (29). 

The  questions  were  answered  in  one  of  three  formats.  One 
was  the  multiple  choice  method,  where  the  respondent  was 
asked  to  choose  from  one  of  the  suggested  responses.  Due 
to  the  possible  unique  aspects  of  DT&E  testing  in  the  dif¬ 
ferent  AFSC  divisions,  an  "other"  choice  was  provided  for 
the  multiple  choice  questions  in  order  to  gain  additional 
information  while  not  forcing  the  respondent  to  choose  an 
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Inaccurate  response.  The  second  method  involved  a  five 
point  Likert  scale  (8:2^8)  which  ranged  from  "strongly 


agree"  to  "strongly  disagree."  A  five  point  scale  was  used 
to  gain  added  sensitivity  while  not  complicating  the  tele¬ 
phone  interview  with  a  large  number  of  choices.  The  final 
method  was  the  open-ended  response  which  was  used  to  gather 
information  which  did  not  lend  itself  to  the  other  two 
methods . 


Demographic  Information.  The  demographic  questions 
wer^^  aimed  at  gathering  data  about  the  program  and  the 
respondent's  background  in  T&E.  The  program  questions  (1  and 
2)  showed  the  types  of  DT&E  testing  done  by  the  program  test 
personnel.  The  background  questions  (30  and  31)  provided  an 
indication  of  the  experience  the  respondent  had  in  T&E. 

Manning  requirements.  This  area  of  the  survey  was 
used  to  determine:  who  accomplished  the  actual  testing  in 
the  program;  where  the  actual  testing  was  done;  and  who 
wrote  the  test  plans  and  procedures.  Questions  3  and  ^ 
represented  the  workload  normally  associated  with  test  man¬ 
agement  duties.  Questions  5  and  6  were  directed  more  toward 
RTO  responsibilities.  Also  implied  in  5  and  6  was  the  pos¬ 
sible  burden  of  TDY.  Analysis  of  negative  responses  to 
question  24,  which  measured  adequacy  of  manning,  with  ques¬ 
tions  3  through  6,  indicated  where  some  of  the  workload 
existed.  Positive  responses  to  24  were  also  matched  with 


responses  to  3  through  6  to  determine  whether  outside  agen¬ 
cies  or  contractors  were  picking  up  more  of  the  load.  Ques¬ 
tions  7  and  8  examined  computer  intensive  programs  similar 
to  simulator  testing. 

Control  of  test  funds.  This  portion  of  the  ques¬ 
tionnaire  was  designed  to  measure  the  effectiveness  of  test 
funds  control  under  the  functional  relationship  between  the 
test  personnel  a.nd  the  program  manager  in  a  self-RTO  program 
Test  managers  were  asked  if  they  had  a  contract-like  agree¬ 
ment  with  the  program  manager.  If  they  did  net,  did  they 
think  It  would  be  helpful?  Questions  10  through  1^  assessed 
the  advantages  or  disadvantages  createa  by  the  program  man¬ 
ager's  direct  control  of  testing  funds  under  the  self-RTO 
concept.  In  an  area  related  to  funds  control,  schedule  con¬ 
trol,  question  23  was  used  to  determine  whe'^her  the  program 
manager  used  his  functional  relationship  with  the  test 
office  to  cut  testing  when  schedule  "slips"  occurred. 

Objectivity.  Objectivity  for  this  survey  dealt  with 
the  "Independent  assessment  of  DT&E  results"  which  is  in  the 
charter  of  an  RTO  (3:6).  It  is  extremely  difficult,  if  not 
impossible,  to  quantify  objectivity.  Therefore,  self-RTO 
objectivity  was  measured  relative  to  independent  RTO  objec¬ 
tivity,  which  was  assumed  to  be  the  standard.  Questions  17 
and  18  measured  whether  or  not  self-RTO  personnel  perceived 
an  independent  RTO  as  providing  more  objectivity  than  a 
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self-RTO,  It  follows,  that  if  objectivity  can  be  increased 
by  replacing  a  self-RTO  with  an  independent  RTO,  the  self- 
RTO  program  was  not  providing  objectivity  up  to  the  standard. 

Objectivity  is  also  implied  in  the  questions  which 
deal  with  program  control.  If  the  test  funds  are  cut  from 
a  self-RTO  program  which  would  not  be  cut  from  an  indepen¬ 
dent  RTO's  funds,  a  loss  of  objectivity  may  be  indicated, 

RTO  assignment.  Questions  19  through  22,  25  and  2? 
dealt  with  RTO  assignment.  They  were  used  to  determine  the 
reasons  the  respondents'  programs  were  under  a  self-RTO  con¬ 
cept.  Analysis  would  show  if  an  independent  RTO  was  consid¬ 
ered  and,  if  so,  why  it  was  not  used.  Is  there  a  reason  for 
the  self-RTO  assignment  in  this  program  or  has  this  program 
office  "traditionally"  been  a  self-RTO? 

Suggested  Improvements.  This  was  an  open-ended 
question  which  gave  the  respondents  an  opportunity  to  recom¬ 
mend  improvements  for  the  self-RTO  program. 

Independent  Test  Organization  Identification 

The  final  research  question  was  answered  by  exam¬ 
ining  mission  statements  of  AFSC  test  facilities  to  identify 
possible  candidates  for  assignment  of  RTO  responsibilities 
for  simulator  testing,  AFSC  Pamphlet  80-27  (^)  summarises 
the  AFSC  test  and  evaluation  ranges  and  facilities  avail¬ 
able  for  AFSC  test  programs.  It  provides  the  test 
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activities'  locations,  missions,  capabilities,  and  some  of 
the  typical  programs  and  functions  each  test  agency  has 
supported  in  the  past.  Since  simulator  DT&E  has  never  been 
performed  by  any  of  the  test  facilities,  the  potential  to 
develop  the  capability  was  also  examined.  The  testing 
requirements  and  resources  determined  from  the  first  research 
question  were  used  to  narrow  the  field  of  APSC  test  organi¬ 
zations.  Once  potential  candidates  were  identified,  they 
were  contacted  to  establish  which  portions,  if  any,  of  a 
typical  simulator  program  they  could  support. 

Summary 

The  research  for  this  project  was  aimed  at  three 
basic  areas.  First,  the  simulator  testing  program  had  to 
be  defined  and  resource  requirements  needed  to  perform  sim¬ 
ulator  testing  had  to  be  determined.  This  was  accomplished 
through  examination  of  a  typical  simulator  program  and  then 
one  resource  requirements  of  the  program  were  defined. 

Second,  a  survey  was  administered  to  identify  the  advantages 
and  disadvantages  of  self-BTO  programs.  Analysis  of  the 
data  was  to  indicate  whether  a  need  exists  throughout  the 
self-RTO  community  to  increase  objectivity,  control  of 
funds,  and  manning.  Also  Incorporated  into  this  survey  was 
a  means  to  identify  how  other  self-RTO* s  are  dealing  with 
these  potential  problems.  Finally,  AFSC  test  facilities 
were  examined  for  capability  to  perform  simulator  RTO 
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responsibilities.  Analysis  of  their  existing  capabilities, 
potential  capabilities,  and  assessment  of  their  ability  to 
function  as  an  independent  RTO  for  simulators  would  allow 
matching  with  the  criteria  established  for  simulator  DT&E. 
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CHAPTER  III 


FINDINDS  AND  ANALYSIS 

Introduction 

This  chapter  presents  the  results  of  the  research. 
The  results  are  presented  in  three  sections;  (a)  Simulator 
Test  Resource  Requirements,  (b)  Self-RTO  Survey,  and 
(c)  Independent  RTO  Identification.  Each  of  these  sections 
corresponds  to  one  of  the  research  questions.  Analysis 
of  the  data  is  summarized  at  the  conclusion  of  each  section 
and  is  used  to  answer  the  research  question. 

Simulator  Resource  Requirements 


Introduction 

The  data  presented  in  this  section  was  gathered  by 
interviewing  personnel  considered  experts  in  the  four 
"unique"  areas  of  simulator  testing  identified  in  Chapter 
II.  The  data  were  analyzed  to  determine  the  criteria  used 
for  success  in  simulator  DT&E  and  what  resources  were  needed 
to  support  testing  in  these  areas. 

Computational  Systems 

The  testing  of  the  computational  system  for  simula¬ 
tors  is  not  as  unique  as  it  has  been  in  the  past.  Most  of 


the  testing  done  on  the  computers  is  done  in  the  offline 
mode;  therefore,  it  is  not  dependent  directly  on  the  simu¬ 
lator.  Mo  special  qualifications  are  required  for  a  compu¬ 
tational  test  person  other  than  a  general  background  in 
computers.  Although  no  courses  are  available  to  train  com¬ 
puter  personnel  specifically  for  simulator  comouter  opera¬ 
tion,  courses  are  available  from  general  purpose  computer 
manufacturers  (6). 

Instructional  Systems 

The  testing  of  instructional  systems  is  a  comclna- 
tion  of  objective  and  subjective  areas.  The  evaluati-'r,  of 
the  instructor's  station  involves  assessment  of  such  items 
as  the  capability  to  track  student  progress  and  vary  the 
workload.  This  area  is  normally  checked  by  monitoring  a 
qualified  aircraft  Instructor  operate  the  instructor's  con¬ 
sole.  Since  the  simulator  is  a  training  device,  it  is 
necessary  to  evaluate  the  capability  of  the  sl.mulator  to 
transfer  training  to  the  student.  The  subjective  nature  of 
this  area  makes  it  difficult  ot  quantify  and,  in  turn,  dif¬ 
ficult  to  design  and  test.  Experience  with  simulator 
inst ruct lonal  techniques  with  a  background  in  engineering 
psychology  and  human  factors  engineering  has  proven  to  be 
useful  In  evaluating  transfer  of  training.  Current  simula¬ 
tor  test  personnel  indicate  such  a  background  is  reeded  to 
facilitate  effective  trade-offs  within  specification  lliT.its 
to  ensure  high  training  results  (11). 
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Mechanoreceptor  Cueing  Systems 

The  hardware  Involved  In  a  Mechanoreceptor  Cueing 
System  (MCS)  may  be  tested  by  a  technician  with  an  engineer¬ 
ing  background  and  an  understanding  of  the  system  provided 
the  test  document  is  well-written.  The  unique  aspect  of 
MCS  testing  is  the  subjective  determination  of  the  am.ount 
of  body  stimulation  needed  to  provide  the  desired  sensation 
of  motion.  The  subtleties  of  how  the  cues  to  the  eyes  work 
with  the  cues  to  the  "seat  of  the  pants"  are  also  Included 
in  visual  system  testing.  An  attempt  must  be  made  to  eval¬ 
uate  all  of  the  cues  available  to  the  student  to  determine 
whether  they  all  provide  a  realistic  sensation  of  motion. 

The  expertise  required  for  MCS  testing  involves  knowing 
what  is  available  in  the  field  and  whar  can  be  expected  for 
state-of-the-art  systems.  Although  the  MCS  hardware  is  the 
easiest  to  develop  and  test,  problems  arise  in  the  defini¬ 
tion  of  what  constitutes  valid  cues  and  how  to  control  the 
device  to  get  the  desired  cues  (7). 

Visual  Systems 

The  basic  requirements  for  a  qualified  test  person 
for  visual  system  testing  include  an  understanding  of: 

1.  the  principles  of  optics; 

2.  how  images  are  formed; 

3.  the  operation  of  test  equipment; 

4.  the  operation  of  related  aircraft  systems; 
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rr'-iiitor  within  specification  limits. 

The  first  four  of  these  areas  may  he  handiel  tv  a  techni¬ 
cian  with  a  background  In  optical  systems  provided  the  test 
plan  was  written  in  sufficient  detail.  However,  the  analy¬ 
sis  portion  of  the  test  draws  on  experience  gair.ec  through 
previous  testing  programs  and  a  knowledge  of  the  general 
design  and  capabilities  of  visual  systems.  For  example, 


visual  system  testing  uses  a  theodilite  to  check  proper 
system  alignm.ent.  This  allgrunent  may  oe  critical  if  tne 
visual  system  is  to  be  interfaced  with  a  highly  accurate 
Heads-Up-Displ ay  (HUD)  weapons  delivery  system.  A  techni¬ 
cian  brought  in  to  perform  this  test  may  be  able  to  set  up 
the  theodilite  and  align  the  HUD  to  obtain  the  required 
data.  The  technician  also  may  be  able  to  complete  the  entire 
test  if  all  test  requirements  meet  the  specification.  The 
problem  arises  when  some  of  the  tests  do  net  meet  all  of 
the  criteria  and  a  decision  must  be  made  as  to  what  is 
acceptable.  For  instance,  will  pushing  the  contractor  to 
full  compliance  in  alignment  cause  a  deterioration  in  reso¬ 
lution?  Experience  in  visual  system  testing  facilitates 
effective  trade-offs  among  these  and  other  decisions  (i5). 


cur 


areas  indicated  that  the  characteristics 


whi  ch 


ulator  testing  apart  from  other  program  testing  appear  to 
be:  (a)  computational  systems;  (b)  instructional  systems; 

(c)  mechanoreceptor  cueing  systems;  and  (al  visual  systems 
In  all  four  areas  the  physical  testing  tasks  were  not  shew 
be  be  extremely  complex,  or  requiring  highly  complicated 
test  equipment.  In  most  cases,  a  test  person  with  a  ger.er 
knowledge  of  the  design  and  operation  of  the  area  being 
tested  could  determine  compliance  'with  specification  -when 
testing  was  accomplished  in  accordance  with  a  well-written 
test  plan. 

The  uniqueness  of  simulator  DT&E  may  be  more  clcS' 
associated  with  the  nature  of  sim.ulator  testing.  Simulate 
testing  deals  with  perception:  (a)  sensations  of  motion; 
(b)  visual  imagery;  and  (c)  student  learning.  Testing  the 
actual  aircraft  can  be  done  by  measuring  actual  rerformiance 
while  simulator  testing  is  based  on  how  well  actual  perfor¬ 
mance  is  approximated.  Perceptions  are  difficult,  if  not 
impossible,  to  quantify.  Ho'wever,  in  order  to  have  an 
effective  simulator  DT4S  program,  testable  specifications 
must  be  written  (11). 

All  four  test  personnel  interviewed  indicated  that 
the  problems  in  simulator  testing  arose  when  deficiencies 
occurred  in  specification  compliance.  In  these  cases,  expe 
rience  in  the  four  unique  areas  became  the  predominant  qual 
iflcatlon  (6;  7;  11;  16).  This  experience  encompasses  the 
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ability  to  recognize  and  implement  possible  trade-offs  with¬ 
in  specification  limits  to  ensure  that  the  delivered  simula¬ 
tor  accurately  approximates  the  aircraft's  systems  and  per¬ 
formance.  The  ability  to  evaluate  and  accept  trade-offs  can 
not  be  written  into  a  test  plan  but  must  be  acquired  through 
exposure  to  simulator  testing  programs. 

A  factor  which  further  complicates  simulator  test¬ 
ing  is  the  relatively  rapid  pace  of  testing  at  the  contrac¬ 
tor’s  facility.  The  changes  to  the  computer  software  which 
drive  the  simulator  can  be  made  quickly.  YWT  personnel  sub¬ 
mit  test  discrepancy  reports  to  the  contractor.  The  con¬ 
tractor  must  attempt  to  bring  the  deficient  areas  within 
specification  tolerances.  A  test  may  be  run  which  uncovers 
discrepancies  which  require  as  many  as  fifty  software 
changes.  The  changes  can  be  made  and  the  test  rerun  the 
next  day.  This  fast-paced,  iterative  process  of  simulator 
testing  requires  an  ability  to  understand  the  impact  of 
changes  on  the  total  system. 

Summary 

The  success  of  a  simulator  DT&E  program  depends  on 
the  ability  to  effectively  test  the  entire  simulator  system. 
The  simulator  contains  four  areas  not  usually  found  in  other 
DT&E  programs:  (a)  computational  systems;  (b)  visual  sys¬ 
tems;  (c)  instructional  systems;  and  (d)  mechanoreceptor 
cueing  systems.  A  qualified  technician  was  determined  to 
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be  capable  of  testing  these  areas  provided  the  technician 
had  a  general  knowledge  of  the  particular  system  and  a  well 
written  test  plan  to  follow. 

The  unique  resource  requirements  were  found  to  be 
the  personnel  with  the  expertise  in  each  of  the  four  test¬ 
ing  areas.  Experience  in  each  one  of  these  area  was  shown 
to  be  the  primary  requirement  when  specification  deviations 
were  encountered  during  testing.  This  experience  facili¬ 
tates  effective  resolution  of  deficiencies  with  the  contrac¬ 
tor.  There  are  no  formal  training  programs  available  to 
build  up  expertise  in  these  four  areas;  therefore,  exper¬ 
tise  is  developed  through  working  with  the  simulator  systems. 

Self-RTO  Survey 


Introduction 

The  data  presented  in  this  section  was  gathered 
using  the  self-RTO  survey  (see  Appendix  A).  The  question¬ 
naire  was  administered  ro  the  test  manager  in  each  program 
office  in  ASD,  ESD,  and  SD  operating  under  the  self-RTO 
concept.  YWT  was  not  Included  in  the  survey.  The  data  were 
analyzed  to  answer  the  second  research  question. 

Demographic  information 

The  median  level  of  experience  of  the  test  managers 
involved  in  self-RTO  programs  was  U  years.  Eight  of  the 
twelve  values  were  6  years  or  less.  Three  of  the  twelve 
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had  some  experience  with  an  independent  RTO  on  another  pro¬ 
gram.  Nine  of  the  twelve  programs  were  Involved  with  a 
combination  of  hardware  and  software  DT&E  while  three  were 
testing  only  hardware. 


Manning 


Table  3-1  indicates  that  75%  of  the  program  offices 


currently  operating  under  the  self-RTO  concept  are  not  per¬ 
ceived  to  be  adequately  manned  to  effectively  perform  both 
test  management  and  RTO  responsibilities.  However,  the 
data  identifies  no  one  area  of  the  manager's  responsibili¬ 
ties  as  the  overriding  cause.  None  of  the  program  offices 
were  responsible  for  complete  preparation  of  test  plans  and 
monitoring  of  the  actual  tests.  Data  indicated  those  pro¬ 
gram  offices  involved  in  preparation  of  test  plans  were  not 
the  ones  sending  personnel  TDY  to  monitor  testing,  but  both 
indicated  the  same  degree  of  inadequate  manning. 

YVT  parallels  the  manning  situation  in  other  pro¬ 
gram  offices  with  RTO  duties.  YWT  is  authorized  26  test 
personnel  and  has  an  actual  manning  of  18.  For  the  most 
part,  simulator  testing  is  accomplished  using  contractor 
prepared  test  plans  and  procedures.  YWT  reviews  the  test 
documents  and  may  be  required  to  provide  inputs  depending 
on  the  experience  of  the  simulator  contractor.  Simulator 
testing  is  also  accomplished  away  from  the  program  office, 
at  the  simulator  contractor's  facility.  However,  YW  is  the 
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only  self-RTO  program  office  where  the  program  office  test 
support  personnel  accomplish  the  actual  testing. 

Control  of  Test  Funds 

Table  3-2  indicates  the  degree  to  which  the  test 
managers  see  the  effectiveness  of  the  functional  relation¬ 
ship  with  the  program  manager  under  self-RTO.  Eighty-three 
percent  of  the  test  managers  agreed  that  the  program  mana¬ 
ger  has  more  control  over  the  test  program  under  the  self- 
RTO  concept  than  under  the  independent  RTO  concept.  Eighty 
three  percent  also  indicated  that  test  funds  were  more  eas¬ 
ily  diverted  away  from  test  and  back  into  the  general  pro¬ 
gram  fund  under  self-RTO  than  independent  RTO,  and  3355  saw 
test  funds  as  the  first  area  usually  cut.  Test  schedules 
were  also  seen  as  an  area  where  program  slips  in  other  part 
of  ^1%  of  the  programs  were  accommodated. 

None  of  the  twelve  self-RTO  programs  had  a  written 
agreement  between  the  program  manager  and  the  test  office 
which  provided  for  testing  tasks  to  be  performed  and  spe¬ 
cific  funds  to  be  used  for  testing.  Table  3-3  shows  67% 
believed  such  a  document  would  be  useful  for  internal  con¬ 
trol  of  test  funds  in  the  program  office  and  58%  considered 
it  potentially  useful  for  control  of  tasks. 

The  data  presented  in  the  funds  control  area  shows 
that  the  program  manager  in  a  self-RTO  program  can,  and 
does,  more  easily  manipulate  test  funds  and  schedules  to 
accommodate  the  entire  program.  The  ability  to  exercise 
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Test  Schedules  Cut  2(17^)  3(25?)  3(25?)  '*(33?) 


TABLE 


Test 

Objectivity  2{\1%)  9(75%)  1(8%) 


more  control  over  the  testing  program  was  perceived  to  en¬ 
hance  the  effective  use  of  program  funds  by  83%  of  the  test 
managers.  However,  67%  of  the  test  managers  considered  an 
Internal,  written  agreement  between  the  program  manager 
and  the  test  manager  helpful  for  management  of  test  funds. 
This  may  indicate  that  the  program  manager,  in  most  cases, 
uses  the  functional  relationship  to  benefit  the  entire  pro¬ 
gram;  however,  test  managers  view  test  funds  as  too  easily 
accessible . 

YWT  has  no  written  agreement  with  program  managers 
to  provide  for  the  type  and  level  of  support  or  the  cost  of 
test  support  requirements.  Although  program  managers  in 
YW  may  be  effectively  using  their  functional  relationship 
with  YWT,  the  survey  data  indicate  a  need  in  self-RTO  pro¬ 
grams  for  formalized  internal  control  of  task  requirements 
and  funds.  This  would  tend  to  provide  more  stability  for 
the  test  funds  and  the  tasks  these  funds- represent .  In 
addition,  test  managers  may  see  a  written  agreement  as  a 
way  to  reduce  potential  conflicts  with  program  managers. 


Ob.1  ectlvlty 

The  data  in  Table  3-^  indicates  97%  of  the  test 
managers  currently  operating  under  self-RTO  believe  that  if 
an  independent  test  agency  were  available,  the  inputs  from 
the  independent  agency  could  increase  the  quality  of  test 
plans  and  enhance  the  objectivity  of  the  entire  test  program. 


The  data  suggests  that  test  plans  under  the  self- 
RTO  concept  have  room  for  Improvement  and  testing  objectlvit 
could  also  be  increased.  The  responses  also  indicate  that 
improvements  could  be  made  with  the  help  of  an  independent 
test  agency. 

RTO  Assignment 

The  reasons  for  assignment  of  RTO  duties  to  the 
program  office  were  found  to  be  in  three  general  areas: 

(a)  high  technology;  (b)  coordination;  and  (c)  schedule  con¬ 
flicts  with  test  organizations.  Five  of  the  program  offices 
contacted  were  assigned  RTO  responsibilities  because  of  a 
high  degree  of  unique  technology.  The  expertise  needed  to 
deal  with  the  testing  question  of  these  programs  resided 
in  the  program  office.  Six  of  the  program  offices  indicated 
that  a  great  deal  of  interface  between  the  program  office 
and  contractor  during  testing  required  close  coordination. 

It  was  determined  by  either  the  program  office  or  AFSC  that 
placing  the  RTO  duties  in  the  program  office  would  facili¬ 
tate  coordination  by  reducing  the  number  of  parties  involved 
in  the  communication  chain.  Finally,  one  program  office 
was  appointed  the  RTO  for  DT&E  testing  because  the  only 
independent  test  facility  capable  of  performing  the  testing 
was  unavailable  because  of  scheduling  conflicts. 

The  self-RTO  method  was  considered  the  only  method 
of  effectively  accomplishing  DT&E  in  83^  of  the  programs. 
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Capable  0  2(17%)  ^(33%)  5(‘^2%)  1(8%) 


Of  th.e  two  that  considered  an  Independent  RTO  an  alternative 
one  was  mentioned  above  as  unable  to  obtain  test  support  due 
to  scheduling  conflicts.  The  other  believed  an  independent 
test  facility  could  handle  the  RTO  duties,  but  anticipated 
some  coordination  problems.  The  four  test  managers  that 
indicated  an  independent  RTO  could  not  test  because  of  an 
inability  to  become  familiar  with  the  design  were  high  tech¬ 
nology  programs.  The  eight  test  managers  who  indicated  a 
test  facility  could  become  familiar  enough  with  the  design 
to  test  it  were  also  the  programs  where  coordination  was 
believed  to  dictate  a  seif-RTO  organization.  Sixty-seven 
percent  of  the  programs  had  requested  to  be  the  RTO.  The 
five  undecided  responses  were  from  test  m.anagers  of  long- 
running  programs.  They  were  not  the  original  test  managers 
of  the  program  and  were  unsure  of  the  original  reasons  for 
RTO  assignment. 

Suggested  Improvements 

The  test  managers  were  asked  to  suggest  improvements 
which  could  be  made  to  their  programs  to  improve  operation 
under  the  self-RTO  concept.  Sixteen  percent  of  the  test 
managers  believed  their  self-RTO  programs  were  operating 
effectively  and  saw  no  needed  improvements.  Fifty  percent 
Indicated  a  need  for  increased  manning  which  paralleled  the 
findings  presented  earlier  under  the  manning  section.  Six¬ 
teen  percent  of  the  test  managers  saw  a  need  for  more 
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experienced  test  personnel.  This  was  brought  out  earlier  in 
the  demographic  section  where  eight  of  the  test  managers  had 
six  or  less  years  of  test  experience.  Finally,  sixteen  per¬ 
cent  believed  an  Independent  test  program  assessment  was 
needed . 

Summary 

Analysis  of  the  data  gathered  using  the  self-RTO 
survey  provided  the  basis  to  answer  the  second  research 
question.  The  responses  indicated  varying  degrees  of  con¬ 
cern  on  the  part  of  test  managers  with  regard  zo  objectiv¬ 
ity,  control  of  test  funds,  and  manning. 

■Manning  in  self-RTO  programs  is  perceived  by  test 
managers  as  Inadequate  to  accomplish  both  test  management 
and  RTO  duties.  It  may  be  unfair  to  single  out  self-RTO 
programs  as  undermanned  during  these  times  of  doing  more 
with  less  throughout  the  Air  Force.  However,  ?5^  of  the 
self-RTO  programs  considered  themselves  undermanned,  YWT 
with  18  of  26  authorizations  filled  is  not  the  exception. 

In  addition  to  the  manning  problem,  YWT  is  the  only  self- 
RTO  program  in  AFSC  with  the  added  responsibility  of  per¬ 
form, Ing  the  actual  testing. 

The  program  manager  appears  to  have  Increased  con¬ 
trol  of  test  funds  under  the  self-RTO  concept.  Righty-three 
percent  of  the  test  managers  see  this  as  a  means  by  which 
program  managers  are  more  effective  in  managing  their 
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However,  the  need  exists  to  establish 


particular  programs, 
more  stability  for  test  funds  and  the  tasks  these  funds 
recresent  by  formalizing  the  funding  process  with  a  written 
agreement  between  the  program  manager  and  the  test  manager. 
An  internal  PID/SOC-type  agreement  could  tie  the  tasks  to 
the  funds.  This  agreement  would  prevent  the  program  mana¬ 
ger  from  redirecting  funds  without  knowing  the  implications 
of  deleting  the  tests  those  funds  represent.  YWT  does  not 
have  a  PID/SOC-type  agreement  within  the  program  office 
for  control  of  test  funds. 

Minety-two  percent  of  the  self-HTO  test  managers 
believe  an  independent  test  organization  could  enhance  the 
quality  of  test  plans  and  procedures  as  well  as  increase 
the  objectivity  of  the  test  program.  However,  tehcnology 
and  coordination  requirements  dictated  the  use  of  tne  self- 
RTO  concept  for  these  programs.  Early  flight  simulation 
dealt  with  new  frontiers  in  technology.  Early  simulator 
testing  required  an  understanding  of  testing  procedures  used 
in  an  area  of  new  technology;  however,  this  uniqueness  now 
deals  more  with  system  expertise  than  with  com.plex  techno¬ 
logical  testing  procedures.  The  current  body  of  simulator 
expertise  resides  within  YW.  This  consolidation  of  simu¬ 
lator  expertise  tends  to  be  the  driving  factor  for  the 
assignment  of  RTO  duties  to  YW. 
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Indeoendent  RTO  Identification 


Introduction 


Cxanination  of  AFSC  test 


(^)  revealed  the  Air  Force  Flight  Test  lenter  (AFFTC)  at 
Edwards  AF5,  California,  as  a  facility  potentially  capatle 
of  ieveloping  a  si.T.ulator  testing  capaoility. 

I”riio  Teso  Wing  has  pilot,  engineer,  and  support 
oapabllir.;,'  to  plan,  conduct,  evaluate,  and  report 
on  TT?tE  of  nannea  and  unmanned  aircraft.  .  .  . 

The  tests  range  from  engineering  simulations  before 
actual  flight  through  envelope  expansion  and  sub¬ 
system  integration  co  flight  tests  of  fully  inte¬ 
grated  v;eaDons  systems  in  a  mission  environment 
r  c  ;  3  '  _ 

AFFTC  personnel  (1^;  18)  were  contacted  to  determine 
what  its  present  and  potential  capabilities  would  be  to 
support  a  simulator  testing  program. 


Background 

AFFTC  has  provided  engineering  support  to  the  Simu¬ 
lator  3P0  on  numerous  simulator  test  programs  and  has  teen 
involved  in  simulator  enhancement  programs  for  the  Tactical 
Air  Command  (TAC). 

TAG  requested  AFFTC  to  analyze  the  F-15  simulator 
to  identify  areas  where  the  F-15  simulator  deviated  from 
actual  aircraft  performance.  AFFTC  personnel  "flew"  the 
simulator  just  as  they  would  fly  the  aircraft  in  a  flight 
test.  Maneuvers  in  the  simulator  were  compared  with  maneu¬ 
vers  in  the  aircraft.  The  discrepancies  were  documented; 
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however,  no  recommendations  were  made  for  correcting  the 
deficiencies . 

TAG  requested  APFTC  to  repeat  the  procedures  used 
on  the  F-15  simulator  on  the  F-i)  simulator.  The  F-A  simu¬ 
lator,  as  it  existed  at  Luke  AFB,  Arizona,  was  brought  up 
on  the  AFFTC  simulator  computer.  The  AFFTC  simulator  "flew" 
with  the  same  deficiencies  as  the  F-4  simulator.  AFFTC 
engineers  went  back  through  all  of  the  aero.natical  c:.rves, 
problem  statements,  and  equations  to  identify  prctlem  areas. 
This  time,  TAG  requested  that  the  discovered  de f i  ;  1  er.c les 
be  corrected.  Ghanges  were  made  to  the  AFFTC  Simula*:  or  me 
correct  discovered  deficiencies.  The  improvements  ;r  fidel¬ 
ity  were  verified,  and  the  changes  were  implemented  into  the 
Luke  F-4  simulator.  TAG  has  now  requested  an  evaluation  of 
the  F-111  simulator  (14). 

Capabilities 

To  date,  AFFTC  has  restricted  its  involvement  in 
simulator  testing  to  the  aerodynamics  of  the  simulation 
model.  AFFTC  test  engl.neers  believe  portions  of  a  weapons 
system  simulate-’  should  be  able  to  be  tested  and  evaluated 
under  ground  rules  similar  to  those  used  in  aircraft  test¬ 
ing.  However,  AFFTC  does  not  possess  the  expertise  needed 
to  accomplish  testing  on  the  visual,  instructional,  compu¬ 
tational,  or  motion  systems  (14;  18 ) . 

AFFTC  is  manned  for  aircraft  testing.  The  added 
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workload  associated  with  simulator  testing  would  degrade 
AFFTC  capabilities  to  test  aircraft  unless  manning  was 
Increased  (18). 

AFFTC  personnel  have  observed  contractual  give-and- 
take  during  simulator  testing.  It  is  believed  that  this 
negotiation  is  more  evident  in  simulator  testing  than  in 
aircraft  testing.  The  concern  is  that  addition  of  another 
party  in  the  simulator  testing  program  may  be  counter¬ 
productive  because  of  added  coordination  problems  (18), 

Summary 

AFFTC  has  the  capability  of  one  day  becoming  the 
RTO  for  simulator  testing.  They  have  recently  increased 
their  involvement  beyond  the  engineering  support  provided 
to  the  Simulator  SPO  to  simulator  troubleshooting  for  TAC. 
Although  these  programs  have  increased  their  exposure  to 
simulator  testing,  they  still  lack  expertise  in  the  test¬ 
ing  areas  identified  as  unique  to  simulator  testing.  Addi¬ 
tion  of  simulator  testing  would  also  create  a  burden  on  the 
AFFTC  operation  unless  Increased  manning  was  provided  or 
priorities  changed. 
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CHAPTER  IV 


SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATION 

Summary 

The  objective  of  the  research  was  to  determine  the 
feasibility  of  assigning  an  independent  Responsible  Test 
Organization  for  development  test  and  evaluation  of  Air 
Force  simulators.  Three  research  questions  were  used  to 
guide  this  study.  The  first  addressed  the  criteria  for 
success  in  simulator  DT&E.  This  o.uestion  was  directed  at 
determining  the  "unique"  characteristics  of  simulator  test¬ 
ing  and  the  resources  required  which  preclude  the  use  :f 
an  Independent  RTO  for  DT&E.  The  second  question  dealt  with 
the  need  for  an  independent  test  organization.  The  final 
research  question  addressed  the  availability  of  an  indepen¬ 
dent  test  organization  capable  of  performing  DT&E  for  simu¬ 
lators.  The  research  was  conducted  through  personal  inter¬ 
views  with  personnel  involved  in  simulator  DT&E,  test  mana¬ 
gers  involved  in  DT&E  under  the  self-RTO  concept,  and  test 
personnel  at  an  independent  test  facility.  Based  on  a 
background  investigation  of  simulator  DT&E  and  data  gathered 
through  the  interviews,  conclusions  were  drawn  and  a  recom¬ 
mendation  for  future  assignment  of  RTO's  for  simulator  DT&E 
was  made. 
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Conclusions 


The  unique  aspects  of  simulator  DT&E  which  led  to 
the  Initial  assignment  of  RTO  duties  to  the  Simulator  SPO 
have  changed.  The  unique  aspects  of  simulator  DT&E  per¬ 
ceived  to  preclude  assignment  of  an  Independent  RTO  are  no 
longer  derived  from  the  Inability  of  an  outside  test  organ¬ 
ization  to  accomplish  the  testing.  The  primary  factor  is 
that  an  independent  tester  would  need  to  develop  expertise 
in  the  four  unique  testing  areas  to  effectively  analyze  and 
deal  with  deficiencies  discovered  during  testing.  Therefore, 
an  independent  test  organization  would  have  to  gradually 
build  up  its  expertise  by  working  with  YW  on  simulator  test 
programs  before  totally  accepting  RTO  responsibility. 

A  substantial  number  of  self-RTO  test  managers  see 
the  need  for  an  independent  test  organization's  added  per¬ 
spective  to  enhance  the  effectiveness  of  the  test  program 
(see  Table  3-4)-  Assignment  of  an  independent  test  organi¬ 
zation  also  complies  more  closely  with  the  intent  of  APR 
80- lA  and  APSC  policy  regarding  assignment  of  RTO  responsi¬ 
bilities  directly  to  the  SPO. 

There  is  a  need  for  an  internal  PID/SOC-type  agree¬ 
ment  between  the  program  manager  and  the  test  manager  in 
self-RTO  programs  (see  Table  3-3)-  This  document  would 
provide  a  mating  of  test  funds  to  test  tasks.  The  program 
manager  would  be  able  to  more  effectively  manage  the  entire 


program  because  test  tasks  would  be  tied  to  test  dollars  and 
the  implications  of  the  tests  lost  by  redirecting  test  funds 
would  be  apparent. 

There  is  an  independent  test  organization  with  an 
expressed  interest  in  simulator  testing  and  an  existing 
capability  to  accomplish  portions  of  a  simulator  test  pro¬ 
gram.  AFPTC  has  provided  engineering  support  on  simulator 
test  programs  managed  by  the  Simulator  SPO.  AFFTC  has  also 
been  involved  in  simulator  enhancement  programs  for  TAG 
which  have  increased  AFFTC 's  expertise  in  the  aerodynamic 
and  computational  aspects  of  simulator  testing.  Therefore, 
AFFTC  is  capable  of  assuming  responsibility  for  the  aerody¬ 
namic  and  computational  portions  of  the  simulator  test  pro¬ 
gram.  Instructional,  visual,  and  mechanoreceptor  cueing 
system  testing  would  be  added  as  AFFTC  gains  experience  in 
testing  these  systems. 

Recommendation 

The  Air  Force  Flight  Test  Center  (AFFTC)  should 
become  the  Responsible  Test  Organization  (RTO)  for  all, 
or  part,  of  the  DT&E  of  Air  Force  simulators.  The  Deputy 
for  Simulators  (ASD/YW)  would  remain  a  source  of  expertise; 
however,  they  would  concentrate  on  test  management  with  RTO 
duties  assigned  to  AFFTC.  An  evolutionary  transfer  of  RTO 
duties  from  YW  to  AFFTC  should  be  used  to  overcome  diffi¬ 
culties  with  expertise  and  manning.  This  evolutionary 


process  would  also  provide  a  relatively  small  amount  of 
risk  because  a  gradual  transfer  of  RTO  responsibilities 
would  not  require  an  abandonment  of  current  practices  for 
a  total  commitment  to  an  unknown  course  of  actiono  YW 
should  also  incorporate  a  PID/SOC-type  document  for  control 
of  test  funds  and  tasks  for  the  remaining  period  of  opera¬ 
tion  as  a  self-RTO. 

Expertise  in  testing  areas  peculiar  to  simulator 
testing  should  be  build  up  gradually  at  APFTC.  Initially, 
AFFTC  should  be  given  responsibility  for  the  aerodynamic 
portions  of  a  simulator  test  program  with  the  RTO  assign¬ 
ment  remaining  in  TW,  APFTC  would  increase  areas  of  testln 
on  subsequent  simulator  programs.  YW  personnel  would  work 
with  and  monitor  AFFTC  personnel  until  AFFTC  testers  were 
capable  of  independently  testing  each  portion  of  the  simu¬ 
lator  system.  For  example,  after  AFFTC  had  completed  the 
aerodynamic  testing  on  a  simulator  program  without  a  YW 
monitor,  another  portion  of  the  simulator  test  program  may 
be  added.  YW  experts  in  the  added  area  would  again  monitor 
and  work  with  AFFTC  until  AFFTC  was  able  to  Independently 
test  this  area.  This  process  would  be  continued  until  the 
entire  simulator  system  is  being  independently  tested  by 
AFFTC.  The  rate  of  take-over  can  be  regulated  by  the  exper 
tise  advances  made  by  APFTC  on  each  simulator  program.  Thi 
would  prevent  an  overload  on  AFFTC  and  possible  degradation 
of  simulator  testing. 
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APPTC  manning  considerations  could  also  be  effec¬ 
tively  handled  under  a  gradual  acceptance  of  RTO  responsi¬ 
bilities.  APPTC  could  not  handle  total  take-over  of  simu¬ 
lator  testing  with  its  current  manning.  The  possibility 
of  manning  authorization  increases  in  the  magnitude  neces¬ 
sary  to  support  simulator  testing  at  APPTC  could  be  diffi¬ 
cult  to  acquire  in  total.  The  evolutionary  process  of  test 
ing  take-over  lends  itself  to  a  gradual  manning  build-up. 
However,  increases  in  APPTC  manpower  would  have  to  be  autho 
rized  by  APSC  as  APPTC  gained  expertise  and  added  new  por¬ 
tions  of  simulator  testing,  or  APPTC  testing  priorities 
would  have  to  be  revised. 

The  responsibility  and  authority  of  APPTC  would 
have  to  be  clearly  delineated  to  lessen  coordination  prob¬ 
lems.  The  rapid  pace  and  iterative  nature  of  simulator 
testing  requires  continual  Interface  with  the  contractor. 

It  is  possible  that  YW  personnel  would  be  present  for 
critical  tests  and,  therefore,  available  for  contractual 
coordination.  However,  APPTC  personnel  would  have  to  be 
given  the  same  authority  to  work  dally  problems  as  has 
been  given  to  YWT  personnel.  This  would  allow  APPTC  to 
work  with  the  contractor  during  testing  to  bring  system 
deviations  within  specification  tolerances  without  exces¬ 
sive  coordination  with  YW.  Any  deviations  outside  of  the 
scope  of  the  contract  which  may  impact  areas  such  as  cost 
and  schedule  would  be  referred  to  YW  for  action. 
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APPENDIX  A 

SELF-RTO  QUESTIONNAIRE 


Instructions 


This  questionnaire  is  designed  to  evaluate  your 
perceptions  of  the  effectiveness  of  assigning  Responsible 
Test  Organization  (RTO)  responsibilities  to  the  program 
office  versus  assigning  RTO  responsibilities  to  an  indepen¬ 
dent  test  organization.  For  purposes  of  this  questionnaire, 
the  phrase  "self-RTO"  will  denote  the  assignment  of  RTO 
responsibilities  directly  to  the  program  office.  The  phrase 
"independent  RTO"  denotes  assignment  of  RTO  responsibili¬ 
ties  to  an  organization  outside  of  the  program  office.  The 
questions  will  be  answered  in  one  of  three  ways.  The  first 
is  multiple  choice,  where  all  responses  will  be  read  after 
the  question  and  you  will  be  asked  to  choose  one  response. 
The  second  is  a  short-phrase  or  one-word  response.  This 
will  normally  be  a  follow-up  question  to  a  multiple  choice 
question.  For  example,  if  the  answer  to  a  previous  ques¬ 
tion  was  "no,"  then  the  follow-up  question  may  address 
"why  not."  The  final  method  for  answering  the  questions 
is  a  five  degree  scale.  A  statement  will  be  read  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree,  or 

E.  Strongly  disagree. 

Your  responses  will  not  be  attributed  to  you  personally  or 
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to  your  office,  but  will  be  used  In  a  data  case  for  statis¬ 
tical  analysis.  If,  at  any  time,  you  would  like  any  of  the 
questions  repeated,  please  ask.  Are  there  any  questions'’ 


RESPOMDEMT 


1.  RTO  responsibilities  for  your  program  are  assigned: 

A.  Directly  to  the  program  office 
3.  To  the  prime  contractor 

C.  To  an  Independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 
A.  Software 

3.  Hardware 

C.  Combination  of  hardware  and  software 

D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by: 
A.  Your  office 
3.  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


j.  The  test  procedures  for  your  program  are  written  by: 

A.  Your  office 

B.  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
?.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done: 
A.  At  the  contractor's  facility 
3.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  dene?) 
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6.  The  actual  DT&E  testing  in  your  program  is  lotie  oy 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 

C.  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 


1.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by; 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

3.  DOD  personnel  assigned  temporarily  to  your  pro¬ 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

E.  Computer  support  not  required  for  my  program 
(Omit  ^*8) 


DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

3.  Have  a  general  knowledge  of  computer  operation, 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

B.  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

3.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  ilsagree. 

If  you  would  1  ixe  '‘he  cues*  ion  or  the  response  scale 
repeated  during  the  questicning,  olease  ask. 


The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  Independent  RTO  concept. 

A  3  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  used  more  effec 
tivelv  than  under  an  independent  RTO  program. 
ABODE 

In  a  self-RTO  program,  DT4E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  use 
in  non-test  areas. 

ABODE 


In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

ABODE 


In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

ABODE 


A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement  normally  signed  by  the  program  office  and  a 
independent  RTO,  would  be  helpful  for  control  of  fund 
under  the  self-RTO  concept. 

ABODE 


A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

ABODE 


Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 
ABODE 


The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  B  0  D  D 


Your  program  requested  to  be  the  RTO  for  your  test 
program. 


An  Independent  RTO  is  a  feasible  alternative  for  year 
test  program, 

ABODE 


An  active  searen  for  an  independent  RTO  foi  your  oro- 
gram  was  conducted  prior  to  your  assignment  as  the  RTO 
ABODE 


There  is  an  Independent  test  organization  capable  cf 
acting  as  the  RTO  for  your  program. 
ABODE 

(If  A  or  3,  what  organization  and  why  Is  it  rot  used' 


23’  The  program  manager  shortens  test  senedu^es  tc  accom¬ 
modate  other  orogram  slios. 

A  B  C  '  D  E 


Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities. 

ABODE 


An  independent  RTO  could  net  become  familiar  encusrh 
with  the  specifications  and  design  of  your  program 
to  effectively  oerform  the  test  program. 

A  5  0  D  E  " 


There  Is  no  difference  in  the  effectiveness  of  the 
control  of  funds  on  a  program  with  an  indeoendent  RTO. 
A  3  0  D  E 


Assignment  of  RTO  responsibilities  to  your  progr 
office  Is  the  only  method  of  effectively  accompl 
your  program's  DT&E  testing. 

ABODE 

(If  D  or  5,  what  other  alternatives  exist  7 ( 


ct  O  ct 


28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 


29.  What  improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 


30.  Have  you  been  involved  in  any  programs  that  used  an 
Independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 

B.  No 


31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 


This  completes  the  questionnaire.  Would  you  like 
0  have  any  questions  repeated?  Do  you  have  any  general 
omments  about  self-RTO  that  were  not  covered  in  the  ques- 
ionnaire? 


APPENDIX  B 
SURVEY  RESULTS 
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RESPONDENT 

1 


1.  RTO  responsibilities  for  your  program  are  assigned; 

Q-  Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2. 


What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

3.  Hardware 

(C).  Combination  of  hardware  and  software 
D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by; 

A.  Your  office 

B.  The  contractor 

C.  Your  office  with  contractor  inputs 
<0)l  The  contractor  with  your  inputs 

S.  Your  office  and  the  contractor  equally 
F.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by; 

A.  Your  office 

B.  The  contractor 

C.  Your  office  with  contractor  inputs 
©  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done; 
©  At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 
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6.  The  actual  DT&E  testing  in  your  program  is  done  by ■ 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 

o  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro 
gram  from  other  computer  resources 

©.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

E.  Computer  support  not  required  for  my  program 
(Omit  ^^8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A,  Are  considered  experts  for  the  parti  ular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

■  B.  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  exnertise  prior  to  program  testing. 

©  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 
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10,  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concent. 

©  B  C  D  ‘  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  ®  C  D  E 


12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 

diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

©BODE 


13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  E 


1^1.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  ©  C  D  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

©BODE 


16.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

©BODE 


17.  Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 
©BODE 


18.  The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 
©  B  C  D  D 
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19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  B  ©  D  E 


20.  An  Independent  RTO  is  a  feasible  alternative  for  your 
test  program. 

A  B  C  D  © 


21.  An  active  search  for  an  Independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO, 
A  B  ©  D  E 


22.  There  is  an  independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  C  D  © 

(If  A  or  B,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

A  B  ©  D  E 


2iJ.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  B  C  ©  E 


25.  An  independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  oerform  the  test  program. 

©  b‘  C  D  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  independent  RTO. 
A  B  ©  D  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
vour  orogram's  DT&E  testing. 

©BODE 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response;  unique  technology 

29.  ’What  Improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  manning 


30.  Have  you  been  involved  in  any  programs  that  used  an 
independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 

(D  No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  6  years 


This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

2 


1,  RTO  responsibilities  for  your  program  are  assigned; 
Directly  to  the  program  office 
Bo  To  the  prime  contractor 

C.  To  an  Independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

B.  Hardware 

©  Combination  of  hardware  and  software 
D.  Other  (Explain) 


3o  The  test  plans  for  your  program  are  written  by: 
A.  Your  office 

The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 
©  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
Fo  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done: 

A.  At  the  contractor's  facility 
H.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

©  Other  (Where  is  the  testing  done?) 

Response:  contractor  and  Air  Force  facilities 
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6.  The  actual  DT&E  testing  in  your  program  is  done ’by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 

©  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro 
gram  from  other  computer  resources 

©  The  contractor 

D.  Other  (Who  provides  computer  support?) 

E.  Computer  support  not  required  for  my  program 
(Omit  #8) 

3.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram; 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

B.  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  orogram  testing. 

©  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have,  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

(B^  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
reoeated  during  the  questioning,  please  ask. 
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The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  Independent  RTO  concept. 

®  B  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  ®  C  D  E 

In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ®  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  E 


In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  (D  C  D  E 


A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concent. 

A  (D  C  D  E 


A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  ©  C  D  E 


Test  plan  inputs  made  by  a  qualifi^ 
could  increase  the  quality  of  t'-  c 
A  B  ©  D 


.cdeoendent  RTO 


The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D  D 
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19. 


Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  (D  C  D  E 

20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  nrogram. 

A  B  C  ®  E 


21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 
A  B  C  ©  E 


22.  There  is  an  independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  ©  D  E 

(If  A  or  3,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slios. 

A  ©  C  D  E 

Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  B  ©  D  E 


25.  An  Independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  oerform  the  test  program. 

A  ©  C  D  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  orogram  with  an  indeoendent  RTO. 
A  B  C  ©  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  orogram's  DT&E  testing, 

^  ■  d)  B  C  D  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  Che  seif-RTO  concept  consiherec  tne  most  effec¬ 
tive  method  of  DT&£  for  your  program'^ 

(Explain'! 

Response:  unique  technology 


29.  What  Improvements  would  you  suggest  to  improve  testing, 
under  the  self-RTO  concept? 

( Explain) 

Response:  manning 


30.  Have  you  been  involved  in  any  programs  that  used  an 
Independent  RTO  for  all  or  part  of  the  DTiE  testing? 

A.  Yes 
No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  17  years 

This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comm.ents  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RTO  resDonsiblllties  for  your  orograr.  are  assigned; 
Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


What  type  of  DT&E  are  you  currently  performing 

A.  Software 
3.  Hardware 

©I  Combination  of  hardware  and  software 


>.  Other  (Explain) 


he  test  plans  for  your  program  are  written  :v 

A.  Your  office 

B.  The  contractor 

C.  Your  office  with  contractor  inputs 
©.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  epuall'.' 
?.  Other  (Who  writes  the  test  plans?) 


The  test  procedures  for  your  program  are  writter 
A.  Your  office 
3.  The  contractor 

C.  Your  office  with  contractor  incuts 
©  The  contractor  with  your  inputs 
E.  Your  office  and  the  contractor  equally 
E.  Other  (Who  writes  the  test  procedures?) 


The  actual  DT&n,  testing  in  your  orogram  is  done: 
©  At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 


i  ^ 


ial  in  your  program  lo  done  ny 

Personnel  r'r'^t:.  your  progra.m  office 
DOD  pernor. nel  located  at  the  test  facility 
Contractor  supplied  personnel 
Other  (Who  doeo  the  actual  tertingl'' 

7  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by- 

A.  DOD  personnel  dedicated  to  your  pr'ogram  f’or- 
comouter  support 

E.  DOD  personnel  assigned  temporarily  to  your  nr 
gram  from  other  com.puter  resources 

C.  The  contractor 

©  Other  (Who  provides  computer  support‘d) 

Response;  independent  contractor  &  S?0  t ere -.nr, 
E.  Computer  supcort  not  required  for  my  pregra.m 
(Omit  .^3)  ’ 

8.  DOD  persor'.nel  used  for  computer  support  r'-.^r  y.nur  pro¬ 
gram; 

A.  Are  considered  experts  for  the  particular 
of  computer  and  associated  programs  prior 
assignment  to  the  test  program, 

3.  Have  a  general  knowledge  of  computer  operation, 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  orogram  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  traini.ng, 

©.  Other  (Explain) 

Response:  phased  in  i  out  during  program 

9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  orovides 
funds  to  be  used  only  for  test  programs'’ 

A,  Yes  (What  type  of  document  is  it?' 

Not  aware  of  any  such  document 

Instructions 


The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

3.  Agree, 

C.  Neither  agree  nor  disa.gree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  q;ae3tion  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 
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The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concent . 

®  3  C  D  "e 


In  a  self-RTO  program,  DT&E  funds  are  used  more  effec 
tively  than  under  an  indeoendent  RTO  program. 

A  ©  C  ■  D  E 

In  a  self-RTO  program,  DT4E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  use 
in  non-test  areas. 

A  3  ©  D  E 


In  a  self-RTO  program,  DT&E  funds  are  more  o?ten  in¬ 
creased  rather  than  decreased. 

A  0  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  ©  C  D  E 


A  contract-liks  agreement,  such  as  a  Memorandum  of 
Agreement  norm.ally  signed  by  the  program  office  and  a 
independent  RTO,  would  be  helpful  for  control  of  fund 
under  the  self-RTO  concent. 

©  3  C  ■  D  E 


A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concent . 

'  ©  3  C  D  E 


3t  plan  Inputs  made  by  a  qualified  independent  RTO 
uild  increase  the  qualitv  of  test  nlans. 

©  3  C  D  E  ’ 


The  perspective  added  by  an  independent  test  agency 
cculd  enhance  the  objectivity  of  the  test  nrogram. 

A  ©  C  D  ''  D 


19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  B  O  D  E 

20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  orogram. 

A  B  C  D  © 


21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  orior  to  your  assignment  as  the  RTO. 
A  B  '  C  D  © 


22.  There  is  an  independent  test  organization  capable  of 
acting  as  the  RTO  for  your  orogram. 

'  A  ©  C  D  E 

(If  A  or  B,  what  organization  and  why  is  it  not  used?! 
Response:  6b95th  SRG ;  coordination  difficulties 

23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

A  B  ©  D  E 


24.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities. 

A  B  C  ©  E 


25.  An  Independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  oerform  the  test  program. 

A  B  C  ®  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  orogram  with  an  independent  RTO. 
A  B  C  ©  E 


27.  Assignm.ent  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  program's  DT&E  testing. 

A  3  C  ©  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  independent  test  agency 


29.  What  improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 


30.  Have  you  been  involved  in  any  programs  that  used  an 
indeoendent  RTO  for  all  or  oart  of  the  DT&E  testing? 
Yes 

B.  No 


31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 


This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

i4 


1.  RTO  responsibilities  for  your  program  are  assigned; 
Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

B.  Hardware 

Combination  of  hardware  and  software 
D.  Other  (Explain) 


3,  The  test  plans  for  your  program  are  written  by: 
©  Your  office 

B.  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


The  test  procedures  for  your  program  are  written  by: 
©  Your  office 

B.  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  procedures?) 


5-  The  actual  DT&E  testing  in  your  program  is  done: 
A.  At  the  contractor's  facility 
©  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D,  At  your  facility 

Ec  Other  (Where  is  the  testing  done?) 
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6.  The  actual  DT&E  testing  in  your  program  is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 
Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 


7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

3.  DOD  personnel  assigned  temporarily  to  your  pro 
gram  from  other  computer  resources 

C.  The  contractor 

(2^  Other  (Who  provides  computer  support?) 

Response:  dedicated  &  temporary  3?0  personnel 

E.  Computer  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

3.  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  program  testing. 

©t  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have,  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  tyoe  of  document  is  it?) 
o.  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

3.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
reoeated  during  the  questioning,  please  ask. 
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10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  indeoendent  RTO  concept. 

®  B  C  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  B  (2)  D  E 

12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ©  C  D  E 

13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

©BODE 

14.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  3  C  ©  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
Independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

A  B  C  ©  E 


l6.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concent . 

A  ©  C  D  E 


17.  Test  plan  Inputs  made  by  a  qualified  Independent  RTO 
could  increase  the  Quality  of  test  plans. 

A  ©  C  D  E 


:.8.  The  perspective  added  by  an  Independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  Q  C  D  D 
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19.  Your  program  requested  to  be  the  RTO  for  your  test 
orogram. 

A  B  ©  D  E 


20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  orogram. 

A  B  C  ©  E 


21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO 
A  B  C  D  © 


22.  There  is  an  independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  ©  D  E 

(If  A  or  3,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

©BODE 


2ii.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  ©  C  D  E 


25 . 


An  independent  RTO  could  not  become  famil 
with  the  specifications  and  design  of  you 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


iar  enough 
r  program 


26.  The''’e  is  no  difference  in  the  effectiveness  of  the 

concrol  of  funds  on  a  program  with  an  indeoendent  RTO. 
A  B  ©  D  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  m.ethod  of  effectively  accomplishing 
your  orogram' s  DT&E  testing. 

A  ©  C  D  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  coordination  difficulties 


29.  What  improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

( Explain ) 

Response:  more  test  training  available 


30.  Have  you  been  involved  in  any  programs  that  used  an 
Independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A .  Yes 
©  No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluat ion? 

Response;  3  years 

This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

5 


1.  RTO  responsibilities  for  your  program  are  assigned: 

©  Directly  to  the  program  office 

B.  To  the  prime  contractor 

G.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


What  type  of  DTStE  are  you  currently  performing? 

A.  Software 

B.  Hardware 

©  Combination  of  hardware  and  software 
D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by: 

A.  Your  office 

B.  The  contractor 

©  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 
3.  The  contractor 

©  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  Inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DTStE  testing  in  your  program  is  done: 

A.  At  the  contractor's  facility 

B.  At  the  user’s  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  ■’’acility 

©  Other  (Where  is  the  testing  done?) 

Response:  combination  of  A,  B,  C,  and  D 


6.  The  actual  DT&E  testing  In  your  program  Is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 

C.  Contractor  supplied  personnel 
Other  (Who  does  the  actual  testing?) 

Response:  combination  rif  a,  E>  and  C 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

3.  DOD  personnel  assigned  temporarily  to  your  pro 
gram  from  other  computer  resources 
C.  The  contractor 

©.  Other  (Who  provides  computer  sappcrt?) 

Response:  contractor  St  DOD  personnel 

E.  Computer  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

@  Are  considered  experts  for  the  particular  tyre 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

3,  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  program  testing. 

C. ,  Have  a  general  knowledge  of  computer  operation 

ana  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

©  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask, 
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10,  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concept. 

©  B  C  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  ©  C  D  E 

12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ©  C  D  E 

13*  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  ©  C  D  E 


1^.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  E  C  ©  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

0  B  C  D  E 

16.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  B  ©  D  E 


17.  Test  :.lan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 

A  ©  C  D  E 


18 .  The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  orogram. 
0  B  C  D  D 
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Yov;r  program 
orogram, 

A 


requested 

@  C 


to  be 
D 


the  R70 


E 


tor  vour 


An  1 

U  6  3  tlr 


hdependent  RTO 
orogram .. 

®  B 


is  a  feasible  alternative  for  your 
C  D  E 


An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  tre  RTO. 
A  ©  C  D  E 


There  is  an  independent  test  orgar.izat ion  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  ©  D  E 

("If  A  or  3,  what  organization  and  why  is  it  not  used?) 


"’he  program  manager  shortens  test  schedules  to  accor,-- 
modate  other  oroeram  slips. 

A  B  ''  C  ©  E 


Curreni  manning  in  your  program  provides  for  effective 
accompl ishment  of  test  management  and  RTO  respcnsl- 
b  i  1  i  t  i  e  3  . 

A  B  C  ©  E 

An  Independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  or'  your  program 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


There  is  no  difference  in  the  effectiveness  or'  the 
control  of  funds  on  a  program  with  an  in  let  ende-’t  RTO. 
A  ©  C  D  S 


Assignment  of  RTO  responsibilities  to  your  pro^rram 
office  is  the  only  method  of  effectively  accomplishing 
your  orogram' 3  DT&S  testing. 

A  ©  C  D  E 

(If  D  or  E,  what  other  alternatives  exist") 
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23.  Why  is  the  self-.^TO  concept  considered  th«  most  efiec- 
tive  method  of  DT&E  for  your  program? 

( Explain ) 

Response;  unique  technology 

29-  What  Improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response;  no  changes  necessary 

30.  Have  you  been  involved  in  any  programs  that  used  an 
indecendent  RTO  for  all  or  oart  of  the  DTiE  testing? 

Yes 

3.  No 

31.  Hovj  m.any  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  13  years 

This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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HESPOMDEriT 
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RTO  responsibilities  for  your  orograrr;  are  assignee: 
Directly  to  the  program  office 
E.  To  the  prime  contractor 

C,  To  an  independent  contractor  hired  for  testing 

D.  Otner  (Who  has  RTO  responsibilities?) 


What  t 


A. 

©t 


C. 


ype  of  DTiE  are  you  currently 
Software 
Hardware 

Combination  of  hardware  and 
Other  (Explain) 


performing? 


so  f tware 


test  plans  for  your  program  are  written  by: 
A.  Your  office 
5,  The  contractor 

O  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  plans?) 


te 

st  procedures 

for  your  program  are  written 

A. 

Your  office 

© 

The  contract 

or 

n 

U  , 

Your  office 

with  contractor  inputs 

D. 

The  contractor  with  your  inputs 

E, 

Your  office 

and  the  contractor  equally 

Other  (Who  writes  the  test  procedures'^' 


by  ; 


The  ac 

tual 

DT&E  testing 

in  your  program  is 

© 

At 

the  contract' 

or's  facility 

B. 

At 

the  user's  facility 

C, 

At 

an  Air  Porce 

test  facility 

D . 

At 

your  facillt 

y 

E  • 

Ot 

her  (Where  is 

the  testing  dene?) 
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6.  The  actual  DT&E  testing  in  your  program  is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 
01  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 


7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro¬ 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

Computer  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

B.  Have  a  general  knowledge  of  computer  operation, 
but  are  assigned  to  the  program  early  enough 

to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have,  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

(§)  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 
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10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concept. 

A  ®  C  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  ®  C  D  E 


12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 

diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ®  C  D  E 


13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  ®  C  D  E 


14.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  B  C  (2)  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
Independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

A  B  ©  D  E  , 


16.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  B  ©  D  E 


17.  Test  plan  Inputs  made  by  a  qualified  Independent  RTO 
could  Increase  the  quality  of  test  plans. 
©BODE 


18.  The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D  D 
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19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  ®  C  D  E 


20.  An  Independent  RTO  Is  a  feasible  alternative  for  your 
test  program. 

A  B  C  ©  E 

21.  An  active  search  for  an  Independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 

A  B  ©  D  E 


22.  There  Is  an  Independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  ©  D  E 

(If  A  or  B,  what  organization  and  why  Is  It  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

A  B  C  ©  E 


24.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities. 

A  B  C  D  © 


25.  An  Independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  Independent  RTO. 
A  B  C  ©  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  program’s  DT&E  testing. 

A  ©  C  D  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  coordination  difficulties 


29.  What  improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  manning 


30.  Have  you  been  involved  in  any  progrsims  that  used  an 
Independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 
©  No 

31.  How  many  years  have  you  been  Involved  in  test  and 
evaluation? 

Response:  1  year 


This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

7 


1.  RTO  responsibilities  for  your  program  are  assigned: 
Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

B.  Hardware 

©l  Combination  of  hardware  and  software 
D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by: 

A.  Your  office 
0  The  contractor 

C.  Your  office  with  contractor  Inputs 

D.  The  contractor  with  your  Inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 
©  The  contractor 

C.  Your  office  with  contractor  Inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done: 
©  At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 
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6.  The  actual  DT&E  testing  in  your  program  is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 
(Si  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 


7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

E.  Computer  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

(BX  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

Not  aware  of  any  such  document 


Instructions 


The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  .you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 
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10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concept. 

A  B  <2)  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  B  ©  D  E 


12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 

diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  B  ©  D  E 


13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  E 


14.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  B  ©  D  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
Independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concent. 

A  B  ©  D  E 


16.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  B  ©  D  E 


17.  Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 

®  B  C  D  E 


18.  The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D  D 


89 


19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  <S)  C  D  E 


20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  program. 

A  B  C  (2)  E 

21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram-  was  conducted  prior  to  your  assignment  as  the  RTO, 

A  (g>  C  D  E 


22.  There  is  an  independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  C  ©  E 

(If  A  or  B,  what  organization  and  why  is  it  not  used?) 

23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

A  ©  C  D  E 


Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities. 

A  B  C  ©  E 


An  Independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

©  B  C  D  E 


There  is  no  difference  in  the  effectiveness  of  the 
control  of  funds  on  a  program  with  an  independent  RTO. 
A  B  ©  D  E 


Assignment  of  RTO  responsibilities  to  your  program 
office  is  the  only  method  of  effectively  accomplishing 
your  program's  DT&E  testing. 

©BODE 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  unique  technology 


29.  What  improvements  would  you  suggest  to  Improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  manning 


30.  Have  you  been  involved  in  any  programs  that  used  an 
independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 
0  No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  2  years 

This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 


RESPONDENT 

8 


RTO  responsibilities  for  your  program  are  assigned: 

0  Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


What  type  of  DT&E  are  you  currently  performing? 

A.  Software 
0  Hardware 

C,  Combination  of  hardware  and  software 

D.  Other  (Explain) 


The  test  plans  for  your  program  are  written  by: 
A.  Your  office 
0  The  contractor 

C.  Your  office  with  contractor  Inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 
0  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  Inputs 

E.  Your  office  and  the  contractor  equally 
F„  Other  (Who  writes  the  test  procedures?) 


The  actual  DT&E  testing  in  your  program  is  done: 
0  At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 
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6.  The  actual  DT&E  testing  in  your  program  is  done  by; 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 

o  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by; 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro¬ 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

©  Computer  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram; 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

B.  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have,  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

©  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 


The  program  marascr  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concept. 

©  B  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program, 

A  <D  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ®  C  D  E 


In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  E 


In  a  self-RTO  program,  DT&E  funds  are  often  the  firsi 
to  be  cut. 

A  B  C  ©  E 


A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

©BODE 

A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  In  a  test  program  under  the  self-RTO 
concept . 

A  ©  C  D  E 


Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 

A  ©  C  D  E 


The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D  D 


19.  Your*  program  requested  to  be  the  RTO  for  your  test 
program. 

A  ®  C  D  E 


20.  An  independent  RTO  Is  a  feasible  alternative  for  your 
test  program. 

A  B  C  ©  E 


21.  An  active  search  for  an  Independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 
©BODE 


22.  There  is  an  independent  test  organization  capable  of 
■  acting  as  the  RTO  for  your  program. 

A  B  C  ©  E 

(If  A  or  B,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

©BODE 

/ 

2^.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  B  C  D  © 


25.  An  independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

A  ©  C  D  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  independent  RTO. 
A  B  C  ©  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accompli.'^hing 
your  program's  DT&E  testing. 

A  ©  C  D  E 

(If  D  or  E,  what  other  alternatives  exist?) 


. 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  unique  technology 


29.  What  Improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  more  experienced  personnel 


30.  Have  you  been  involved  in  any  programs  that  used  an 
Independent  RTO  for  all  or  part  of  the  DT&E  testing? 
A.  Yes 
©  No 


31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  30  years 


This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

9 


1.  RTO  responsibilities  for  your  program  are  assigned: 

(S)t  Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

B.  Hardware 

©t  Combination  of  hardware  and  software 
D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by: 

A.  Your  office 

The  contractor 

C.  Your  office  with  contractor  Inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 
©  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done: 
©  At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 
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6.  The  actual  DT&E  testing  in  your  program  Is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 
(D  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro¬ 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

E.  Computer  support  not  required  for  my  program 
(Omit  if 8) 

8.  DC"  personnel  used  for  computer  support  for  your  pro- 
gram: 

Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

B.  Have  a  general  knowledge  of  computer  operation, 
but  are  assigned  to  the  program  early  enough 

to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

(2)l  Not  aware  of  any  such  document 


Instructions 


The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 
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10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  Independent  RTO  concept. 

A  B  ©  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  ©  C  D  E 

12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ®  C  D  E 


13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  C  ©  E 


lU.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  ©  C  D  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

A  B  C  ©  E 


16.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  B  C  ©  E 


17.  Test  plan  Inputs  made  by  a  qualified  Independent  RTO 
could  Increase  the  quality  of  test  plans. 
©BODE 


18.  The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D  D 
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19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  ©  C  D  E 

20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  program. 

A  B  C  (2)  E 

21.  An  active  search  fox’  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 

A  B  C  ©  E 


22.  There  is  an  Independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  C  ©  E 

(If  A  or  B,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  prograjn  slips. 

A  ©  C  D  E 


2^.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  B  C  ©  E 


25.  An  independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  Independent  RTO. 
A  B  C  ©  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  program’s  DTiE  testing. 

A  ©  C  C  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  coordination  difficulties 


29.  What  improvements  would  you  suggest  to  Improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  manning 


30,  Have  you  been  involved  in  any  programs  that  used  an 
independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 

<g)l  No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  3  years 


This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

10 


1.  RTO  responsibilities  for  your  program  are  assigned: 
Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 

A.  Software 
©  Hardware 

C,  Combination  of  hardware  and  software 

D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by: 
A.  Your  office 
01  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 
©  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 
P.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done: 
At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Porce  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 
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6. 


The  actual  DT&E  testing  in  your  program  is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 
o  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 


7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro¬ 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

o  Comouter  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

B.  Have  a  general  knowledge  of  computer  operation, 
but  are  assigned  to  the  program  early  enough 

to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

(5^  Not  aware  of  any  such  document 


Instructions 


The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 
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10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  independent  RTO  concept . 

A  d)  C  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  ®  C  D  E 

12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  ®  C  D  E 


13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  E 


14.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  ®  C  D  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

A  <g)  C  D  E 


16.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  B  C  ©  E 


17.  Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  Increase  the  quality  of  test  plans. 

A  ®  C  D  E 


18.  The  perspective  added  by  an  independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D  D 


_ _ X 


■  ■  ■> — 
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19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  B  ©  D  E 


20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  program. 

A  B  C  ©  E 


21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 
A  B  C  ©  E 


22.  There  is  an  independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  C  ©  E 

(If  A  or  B,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

ABC  ©  E 


24.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  B  C  ©  E 


25.  An  independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  independent  RTO. 
A  B  C  ©  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  program's  DT&E  testing. 

A  ©  C  D  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  coordination  difficulties 


29.  What  improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  no  changes  necessary 


30.  Have  you  been  involved  in  any  programs  that  used  an 
independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 
<§>  No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  4  years 


This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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RESPONDENT 

11 


1.  RTO  responsibilities  for  your  program  are  assigned: 
Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


2.  What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

B.  Hardware 

©  Combination  of  hardware  and  software 

D.  Other  (Explain) 


3.  The  test  plans  for  your  program  are  written  by: 

A.  Your  office 

B.  The  contractor 

©  Your  office  with  contractor  Inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


4.  The  test  procedures  for  your  program  are  written  by: 

A.  Your  office 

B.  The  contractor 

©  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  procedures?) 


5.  The  actual  DT&E  testing  in  your  program  is  done: 

A.  At  the  contractor's  facility 

B.  At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

©  Other  (Where  is  the  testing  done?) 

Response:  combination  of  A,  B,  C,  and  D 
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6.  The  actual  DT&E  testing  in  your  program  Is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 
©  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

E.  Computer  support  not  required  for  m.y  program 
(Omit  #8) 


8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram; 

@  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program, 

B,  Have  a  general  knowledge  of  computer  operation 
but  are  assigned  to  the  program  early  enough 
to  gain  expertise  prior  to  program  testing, 

C,  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training, 

D,  Other  (Explain) 


9,  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

©  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale,  A  statement  will  be  read  to  you  and  you 
v/ill  be  asked  whether  you: 

A,  Strongly  agree, 

3,  Agree, 

C,  Neither  agree  nor  disagree, 

D,  Disagree, 

E,  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask, 
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10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  Independent  RTO  concept. 

0  B  C  D  E 


11.  In  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  0  C  D  E 

12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  program  fund  to  be  used 
in  non-test  areas. 

A  0  C  D  E 


13.  In  a  self-RTO  program,  DT&E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  E 


14.  In  a  self-RTO  program,  DT&E  funds  are  often  the  first 
to  be  cut. 

A  B  C  0  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement  normally  signed  by  the  program  office  and  an 
independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

A  ©  C  D  E 


16.  A  contract-like  agreement,  such  as  a  Memorandum  of 

Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concent . 

0  B  C  D  E 


17.  Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 

0  B  C  D  E 


18.  The  perspective  added  by  an  Independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  B  ©  D  D 
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19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  ®  C  D  E 

20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  program. 

A  ©  C  D  E 


21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 
A  ©  C  D  E 


22.  There  is  an  Independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  ©  C  D  E 

(If  A  or  3,  what  organization  and  why  is  it  not  used?) 
Response:  32^6th  TW;  schedule  conflicts 

23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

A  B  C  ©  E 

2h,  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities. 

A  ©  C  D  E 


25.  An  independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  Independent  RTO. 
A  ©  C  D  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  program's  DT&E  testlr^. 

A  B  C  ©  E 

(If  D  or  E,  what  other  alternatives  exist?) 

Response:  independent  test  facility 
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28.  Why  is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  test  facility  schedule  problems 


29.  What  improvements  would  you  suggest  to  improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  independent  assessment  needed 


30.  Have  you  been  involved  in  any  programs  that  used  an 
independent  RTO  for  all  or  part  of  the  DT&E  testing? 

(S)  Yes 
B.  No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  22  years 

This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 


RESPONDENT 

12 


RTO  responsibilities  for  your  program  are  assigned: 

<s>  Directly  to  the  program  office 

B.  To  the  prime  contractor 

C.  To  an  Independent  contractor  hired  for  testing 

D.  Other  (Who  has  RTO  responsibilities?) 


What  type  of  DT&E  are  you  currently  performing? 

A.  Software 

B.  Hardware 

©  Combination  of  hardware  and  software 
D.  Other  (Explain) 


The  test  plans  for  your  program  are  written  by: 
A.  Your  office 
©  The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  plans?) 


The  test  procedures  for  your  program  are  written  by: 
A.  Your  office 

The  contractor 

C.  Your  office  with  contractor  inputs 

D.  The  contractor  with  your  Inputs 

E.  Your  office  and  the  contractor  equally 

F.  Other  (Who  writes  the  test  procedures?) 


The  actual  DT&E  testing  in  your  program  is  done: 
A.  At  the  contractor's  facility 
At  the  user's  facility 

C.  At  an  Air  Force  test  facility 

D.  At  your  facility 

E.  Other  (Where  is  the  testing  done?) 


6.  The  actual  DT&E  testing  in  your  program  is  done  by: 

A.  Personnel  from  your  program  office 

B.  DOD  personnel  located  at  the  test  facility 

©  Contractor  supplied  personnel 

D.  Other  (Who  does  the  actual  testing?) 

7.  If  required,  computer  support  personnel  for  your  pro¬ 
gram  testing  are  supplied  by: 

A.  DOD  personnel  dedicated  to  your  program  for 
computer  support 

B.  DOD  personnel  assigned  temporarily  to  your  pro¬ 
gram  from  other  computer  resources 

C.  The  contractor 

D.  Other  (Who  provides  computer  support?) 

©  Computer  support  not  required  for  my  program 
(Omit  #8) 

8.  DOD  personnel  used  for  computer  support  for  your  pro¬ 
gram: 

A.  Are  considered  experts  for  the  particular  type 
of  computer  and  associated  programs  prior  to 
assignment  to  the  test  program. 

B.  Have  a  general  knowledge  of  computer  operation, 
but  are  assigned  to  the  program  early  enough 

to  gain  expertise  prior  to  program  testing. 

C.  Have  a  general  knowledge  of  computer  operation 
and  require  no  further  special  training. 

D.  Other  (Explain) 


9.  Does  your  program  have  a  written  agreement  between  the 
program  manager  and  the  test  personnel  which  provides 
funds  to  be  used  only  for  test  programs? 

A.  Yes  (What  type  of  document  is  it?) 

©  Not  aware  of  any  such  document 


Instructions 

The  following  questions  will  be  answered  using  the 
five  point  scale.  A  statement  will  be  read  to  you  and  you 
will  be  asked  whether  you: 

A.  Strongly  agree, 

B.  Agree, 

C.  Neither  agree  nor  disagree, 

D.  Disagree, 

E.  Strongly  disagree. 

If  you  would  like  the  question  or  the  response  scale 
repeated  during  the  questioning,  please  ask. 


10.  The  program  manager  has  more  direct  control  of  the 
test  program  funds  under  the  self-RTO  concept  than 
under  the  Indeoendent  RTO  concept. 

A  (D  C  D  E 


11.  Ir.  a  self-RTO  program,  DT&E  funds  are  used  more  effec¬ 
tively  than  under  an  independent  RTO  program. 

A  0  C  D  E 

12.  In  a  self-RTO  program,  DT&E  funds  are  more  easily 
diverted  back  into  the  general  progra.m  fund  to  be  used 
in  non-test  areas. 

A  <B)  C  D  E 


13-  In  a  self-RTO  program,  DT4E  funds  are  more  often  in¬ 
creased  rather  than  decreased. 

A  B  ©  D  £ 


1-1.  In  a  self-RTO  program,  DTiE  funds  are  often  tne  first 
to  be  cut. 

A  3  ©  D  E 


15.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement  normally  signed  by  the  program  office  and  an 
Independent  RTO,  would  be  helpful  for  control  of  funds 
under  the  self-RTO  concept. 

'  A  ©  C  D  E 

16.  A  contract-like  agreement,  such  as  a  Memorandum  of 
Agreement,  would  be  helpful  for  control  of  the  tasks 
to  be  performed  in  a  test  program  under  the  self-RTO 
concept . 

A  ©  C  D  E 


17.  Test  plan  inputs  made  by  a  qualified  independent  RTO 
could  increase  the  quality  of  test  plans. 

A  ©  C  D  E 


l3.  The  perspective  added  by  an  Independent  test  agency 
could  enhance  the  objectivity  of  the  test  program. 

A  ©  C  D 


nil 


j 


19.  Your  program  requested  to  be  the  RTO  for  your  test 
program. 

A  <g)  C  D  E 

20.  An  independent  RTO  is  a  feasible  alternative  for  your 
test  program. 

A  B  C  0  E 

21.  An  active  search  for  an  independent  RTO  for  your  pro¬ 
gram  was  conducted  prior  to  your  assignment  as  the  RTO. 

A  B  ©  D  E 


22.  There  is  an  Independent  test  organization  capable  of 
acting  as  the  RTO  for  your  program. 

A  B  C  ©  E 

(If  A  or  B,  what  organization  and  why  is  it  not  used?) 


23.  The  program  manager  shortens  test  schedules  to  accom¬ 
modate  other  program  slips. 

A  B  ©  D  E 


24.  Current  manning  in  your  program  provides  for  effective 
accomplishment  of  test  management  and  RTO  responsi¬ 
bilities  . 

A  B  C  ©  E 


25.  An  independent  RTO  could  not  become  familiar  enough 
with  the  specifications  and  design  of  your  program 
to  effectively  perform  the  test  program. 

A  B  C  ©  E 


26.  There  is  no  difference  in  the  effectiveness  of  the 

control  of  funds  on  a  program  with  an  independent  RTO. 
A  B  C  ©  E 


27.  Assignment  of  RTO  responsibilities  to  your  program 

office  is  the  only  method  of  effectively  accomplishing 
your  program's  DT&E  testing. 

A  ©  C  D  E 

(If  D  or  E,  what  other  alternatives  exist?) 
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28.  Why  Is  the  self-RTO  concept  considered  the  most  effec¬ 
tive  method  of  DT&E  for  your  program? 

(Explain) 

Response:  coordination  difficulties 


29.  What  Improvements  would  you  suggest  to  Improve  testing 
under  the  self-RTO  concept? 

(Explain) 

Response:  manning 


30.  Have  you  been  involved  in  any  programs  that  used  an 
independent  RTO  for  all  or  part  of  the  DT&E  testing? 

A.  Yes 
No 

31.  How  many  years  have  you  been  involved  in  test  and 
evaluation? 

Response:  4  years 

This  completes  the  questionnaire.  Would  you  like 
to  have  any  questions  repeated?  Do  you  have  any  general 
comments  about  self-RTO  that  were  not  covered  in  the  ques¬ 
tionnaire? 
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to  an  independent  test  organization.  The  Air  Force  Flight  Test 
Center  at  Edwards  AFB  CA  was  identified  as  a  test  organization 
capable,  or  potentially  capable,  of  conducting  all,  or  part,  of 
the  Air  Force  simulator  DT&E  program. 
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